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EXECUTIVE SUMMARY 


The National Assessment Governing Board is an 
independent, bipartisan organization that sets policy 
for the National Assessment of Educational Progress 
(NAEP). The Governing Board established the 
NAEP Program of 12th Grade Preparedness 
Research to assess what NAEP can report on the 
academic preparedness of 12th grade students 
entering college and job training. The Governing 
Board commissioned the Educational Policy 
Improvement Center in October 2012 to conduct the 
College Course Content Analysis (CCCA) study, 
which used course artifacts to identify the prerequisite 
knowledge, skills, and abilities (KSAs) in mathematics 
and reading that are necessary for students to be 
prepared to qualify for entry-level, credit-bearing 
courses that satisfy general education requirements. 

The CCCA study was designed to answer the 
following research questions: 

• What are the prerequisite KSAs in 
mathematics and reading to qualify for entry- 
level, credit-bearing courses that satisfy 
general education requirements? 

• Plow do these prerequisite KSAs compare 
with the 2009 and 2013 NAEP mathematics 
and reading frameworks and item pools? 

• Plow do these prerequisite KSAs compare 
with previous NAEP preparedness research, 
i.e., descriptions of minimal academic 
preparedness requirements produced in the 
Judgmental Standard Setting (JSS) research? 

• Plow can these prerequisites inform future 
NAEP preparedness research? 


Study Method 

Phase I : Establishing the Foundation for the Study 
The initial phase included selecting a representative 
sample of institutions, collecting course artifacts, 
organizing the artifacts into course packets, and 
recmiting content reviewers and NAEP content 
experts. 

Selecting a representative sample. One hundred 
fifty-one (151) institutions contributed 160 course 
packets, which included a syllabus, textbook, and class 
assignment or assessment (hereafter referred to as 
course artifacts). Plalf of the 160 packets addressed 
mathematics courses, and the other half addressed 
courses with substantial reading demands. To 
generate a nationally representative sample of 
institutions, the target sample was stratified by 

• program type (two-year and four-year) 

• size (small < 4,999 students; medium 5,000- 
9,999; or large > 10,000) 

• geographic region (East or West) 

• institutional control (private or public) 

Collecting course artifacts and assembling course 
packets. The mathematics course titles included 
precalculus/calculus, college algebra, finite 
mathematics, and statistics. The courses with 
substantial reading demands included English 
literature, psychology, U.S. government, and U.S. 
history. Instmctors who taught the included courses 
at the sampled institutions provided the course 
artifacts and verified that these artifacts met inclusion 
criteria. The course artifacts were compiled into 
course packets for the content analysis activities. 




Recruiting content reviewers and NAEP content 
experts. Highly qualified college-level content 
experts, 16 in mathematics and 16 in reading, were 
recmited to conduct course content reviews. The 
content reviewers were postsecondary instructors 
who taught relevant introductory college courses. 
Another smaller group of doctoral-level content 
experts in mathematics and reading with specialized 
expertise and experience with the NAEP frameworks 
were engaged to provide guidance for the use of the 
NAEP frameworks within the overall study design. 

Phase 2: Conducting Content Analysis Activities 
The next phase included training and qualifying the 
content experts to conduct two content reviews, 
independent and group, as part of the convergent 
consensus process; summarizing the data into detailed 
content maps of the KSAs; and completing a 
generalizability study, as a measure of the consistency 
of the coding process. 

Training and qualifying reviewers. The training 
began by inviting content reviewers to conduct a 
“holistic” review of the course packets prior to being 
introduced to the NAEP framework. This provided 
reviewers with the opportunity to identify prerequisite 
KSAs independent of NAEP frameworks. 

Subsequent reviewer trainings were conducted to 
address the NAEP framework, the coding scheme, 
decision rules, and review processes. Training also 
included trial reviews of a subset of the course 
packets and individualized feedback for each 
reviewer. Only reviewers who demonstrated an 
understanding of the materials and processes qualified 
to participate in the content reviews. 

Independent and group content reviews. After 
training, the research project moved into the 
operational components of the study by facilitating 
two content expert reviews of the course packets: one 
independent review where reviewers individually 


applied a comprehensive coding scheme to the course 
packets, and one group review where the reviewers 
discussed their points of individual agreement and 
disagreement. Content reviewers also used a set of 
decision mles to support their identification of the 
prerequisite KSAs within each course packet. 

Decision rules are clarifying guidelines for pre- 
identified potential areas of ambiguity. 

Summarizing the data and conducting reliability 
checks. The data collected from these reviews were 
summarized into content maps of the prerequisite 
KSAs for each course. Some prerequisite KSAs were 
not included within the NAEP framework objectives 
but were evident in the course packets. A 
generalizability study was conducted to evaluate 
coding reliability. 

Phase 3: Conducting NAEP Expert Reviews 
During the final phase, the NAEP experts used the 
content maps to develop narrative descriptions of the 
prerequisite KSAs necessary for students to be 
prepared for entry, without remediation, into credit- 
bearing entry-level courses. They then compared their 
narrative descriptions to NAEP item pools, 
borderline performance descriptions generated in 
previous research, and other content summaries 
captured in the Judgmental Standard Setting and Job 
Training Program Content studies. In these 
comparisons, NAEP experts analyzed 
correspondence between the CCCA KSAs and 
content assessed in the NAEP. Through a facilitated 
discussion, they provided insight via their NAEP 
framework expertise. 

Study Process Results 

The validity of the results is based on institutional 
representativeness, artifact sufficiency, and coding 
reliability. Analysis indicates that the sample of 
institutions submitting artifacts for courses with 
substantial reading demands is generally 


representative of the population of institutions. 
Institutions submitting artifacts for mathematics 
courses underrepresent small institutions and private 
institutions and overrepresent public institutions and 
large institutions. Post hoc analyses suggest that 
deviations from representativeness do not pose a 
serious threat to validity. Content reviewers deemed 
the artifacts sufficient to allow them to focus on what 
students are expected to be able to learn, what 
students are expected to be able to do to demonstrate 
learning, and the kinds of content the students will 
reference in order to learn. Results from the 
generalizability study indicate the CCCA findings are 
reliable, i.e., rating was consistent within course 
packets during the independent and group reviews. 

Prerequisite KSA Findings and 
Conclusions 

Prerequisite KSAs 

The results from the study indicate that most of the 
prerequisite KSAs for both mathematics courses and 
courses that require extensive college-level reading are 
reflected in the NAEP frameworks. A KSA was 
considered prerequisite if a student is either expected 
or required to possess this knowledge, skill, or ability 
to be prepared for entry into the course. The 
prerequisite KSAs were mapped to the NAEP 
frameworks or they were included as non-NAEP 
additional KSAs. Any KSA identified in 75% or more 
of course packets within a subject area was considered 
to be a common prerequisite. Prerequisite KSAs 
found less frequently within course packets 
demonstrated the range of prerequisites present 
within the sample of course packets, and, by 
extension, likely to be present within courses taught 
across U.S. institutions. 

Reading prerequisite KSAs. Some of the reading 
KSAs that are prerequisite to entering entry-level, 
credit-bearing college courses have substantial 
informational text reading demands, and some of the 


reading KSAs demand engagement with literary texts. 
KSAs common to both included the ability to locate 
or recall textually explicit information within and 
across texts and the ability to take different 
perspectives in relation to a text. Few differences 
were found in prerequisite KSAs among courses with 
substantial informational text reading demands. The 
key distinction between the courses with substantial 
informational text versus literary text reading 
demands was in the KSAs related to the ability to 
make complex inferences within and across texts; 
related KSAs were more commonly found to be 
prerequisite in courses dependent on informational 
texts rather than literary texts. However, courses 
within English literature were more likely than those 
with substantial informational text reading demands 
to have prerequisite KSAs relating to both 
informational and literary texts. 

Mathematics prerequisite KSAs. The mathematics 
KSAs were mosdy specific to a course title, i.e., there 
was not a single set of prerequisite KSAs that covered 
all entry-level, credit-bearing courses. The majority of 
prerequisite KSAs were represented in objectives 
within the NAEP framework’s Numbers and Operations 
subject area. Prerequisite KSAs for 
precalculus/ calculus and college algebra were notably 
different from those for finite mathematics and 
statistics — they were represented by the objectives 
within the 1 Variables, Expressions, and Operations and 
Equations and Inequalities subtopics of the Algebra 
subject area of the NAEP framework, whereas there 
were fewer prerequisite KSAs from these subject 
areas for statistics and finite mathematics. Additional 
KSAs that were not found within the NAEP 
framework were identified as prerequisites. These 
related to the use of technology (e.g., calculators, 
online resources) and the ability to read and 
communicate in various modes about mathematics. 


Comparison of Prerequisite KSAs to 2009 and 
20 1 3 NAEP Mathematics and Reading Frameworks 
and Item Pools 

Reading prerequisite KSA comparison. The NAEP 
reading framework aligned well with the prerequisite 
KSAs identified in this study, especially in the cognitive 
domains of Yjocate! Recall and Integrate / Interpret. The 
prerequisite KSAs focused less on the ability to critique 
or evaluate, which may be what is covered within a 
college course and therefore not a prerequisite to entry 
into the course. The reading item pools represent fairly 
well the prerequisite KSAs noted in this study. The 
nature of the text excerpts in the 2009 and 2013 
NAEP reading item pools did not, however, cover the 
range of texts that would be required in order to assess 
all of the prerequisite KSAs. 

Mathematics prerequisite KSA comparison. The 

“big picture” of the mathematics frameworks was 
transparent to the NAEP experts when they 
envisioned the 4th, 8th, and 12th grade objectives side 
by side. After accounting for the pieces of NAEP 
objectives that were not considered relevant, some 
prerequisite KSAs are found in the 8th grade NAEP 
framework. Some additional KSAs identified by 
content reviewers were found by the NAEP experts 
to be implied within the NAEP frameworks and 
explicit in the frameworks for grades not referenced 
in this study. The CCCA prerequisite KSAs focused 
more heavily on application of mathematics (e.g., 
simulations, addressing real-world problems) than is 
evident in the item pools. The NAEP experts found 
the items to be skill-based and procedural in nature 
rather than applied. The CCCA content focused on 
application problems, which do match the NAEP 
framework, but the items do not reflect the same level 
of focus. Another main difference between NAEP 
mathematics and CCCA prerequisite KSAs is that 
many of the CCCA sampled courses did not have 
measurement or geometry objectives as prerequisite 
KSAs. 


Comparison of Prerequisite KSAs to Previous 
NAEP Preparedness Research 
Reading prerequisite KSA comparison. The CCCA 
prerequisite KSAs describe a depth of understanding 
and a level of cognitive demand that are beyond those 
described by the JSS minimum preparedness 
requirements. The NAEP experts portrayed the two 
sets of descriptions as concentric circles, with the JSS 
circle completely inside the CCCA circle. They 
concluded that the CCCA study, with the inclusion of 
the courses that relied heavily on informational texts 
as well as those relying primarily on literary texts, 
resulted in a narrative description that was broader 
and deeper than the JSS borderline performance 
descriptions (BPD). 

The reading prerequisites identified in the CCCA 
study are more numerous and constitute more KSAs 
related to higher-order thinking skills than those 
described by the JSS descriptions of minimal 
preparedness. The specific NAEP objectives 
necessary for minimal preparedness from JSS were all 
subsumed by the prerequisites evident in CCCA 
course artifacts. For example, the JSS description 
included offer evidence in support op but not judging, 
evaluating, and critiquing, which is subsumed by 
offering evidence to support a claim. 

Mathematics prerequisite KSA comparison. The 

prerequisites identified for precalculus / calculus and 
college algebra are similar to the JSS description of 
the KSAs necessary for minimal preparedness for 
entry-level college mathematics courses. The 
prerequisites for finite mathematics and statistics, 
however, require fewer and less cognitively complex 
KSAs than are described by the JSS description. 

Given the differences in evidence, process, and unit 
of analysis, dissimilarities between study findings were 
expected. The CCCA study identified prerequisite 
KSAs from the evidence contained in college course 


packets of artifacts. As a standard-setting process, the 
JSS study identified what minimally prepared college 
students need to know and be able to do to succeed 
in entry-level college mathematics courses and 
courses with substantial reading demands. 

Informing Future NAEP Preparedness Research 

Studies focusing on the prerequisites from a broader 
range of entry-level college courses might be useful at 
identifying differences in prerequisites for courses 
that use a variety of texts and text types. Of particular 


interest is the inclusion of courses that contain both 
significant mathematics and reading demands (e.g., 
economics) and the examination of the prerequisites 
related to both content areas. 

Exploring prerequisites related to writing, graphical 
representations of information, and technology use 
could complement the content analyses of reading 
and mathematics KSAs that have been conducted on 
college and job training courses. 


INTRODUCTION 


This report describes the College Course Content 
Analysis (CCCA) study, one of a series of studies 
contributing to the National Assessment of 
Educational Progress (NAEP) Program of 12th 
Grade Preparedness Research sponsored and 
overseen by the National Assessment Governing 
Board, hereafter referred to as the Governing Board. 
The Governing Board is an independent, bipartisan 
organization authorized and funded by Congress to 
set policy and provide general oversight and direction 
for NAEP, commonly known as The Nation’s Keport 
Card. NAEP is the largest nationally representative 
and continuing assessment of what 4th, 8th, and 12th 
grade students know and can do in 12 academic 
subject areas. 


NAEP Program of 12th Grade 
Preparedness Research 

Since 2004, the Governing Board has been working to 
determine the feasibility of NAEP reporting on the 
academic preparedness of 12th grade students, relative 
to postsecondary education and job training programs. 
The Governing Board convened a seven-member 
technical panel in 2007 to assist in the planning of 
research and validity studies to determine how and 
what NAEP could report on the preparedness of 12th 
grade students (WestEd, 2010). The Technical Panel 
on 12th Grade Preparedness Research recommended a 
multimethod approach comprised of five separate 
areas of research, each targeting a major research 
question, as shown in Figure 1 . 



Figure I. NAEP’s Program of Preparedness Research 




The Governing Board commissioned the Educational 
Policy Improvement Center (EPIC) to conduct a 
study to determine the relationship between the 
prerequisite knowledge, skills, and abilities (KSAs) in 
reading and mathematics that students need for entry- 
level college courses and the 12th grade NAEP 
reading and mathematics assessment content. 

Artifacts from entry-level college courses were 
reviewed to determine the KSAs necessary for 
students to be academically prepared for entry into 
these courses and the degree to which the 12th grade 
NAEP reading and mathematics frameworks aligned 
with the necessary prerequisite KSAs. 

The CCCA study was designed to produce research 
findings that contribute to the overall NAEP 
Program of 12th Grade Preparedness Research and 
that can serve as validity evidence in relation to 
interpretations of the results from two previous 
preparedness studies — the Judgmental Standard 
Setting (JSS) study, implemented in 2011, and the Job 
Training Program Curriculum (JTPC) study, 
implemented in 2012. The JSS study followed a 
modified bookmarking, standard-setting process to 
identify the NAEP objectives that represented the 
mathematics and reading knowledge, skills, and 
abilities prerequisite to occupational job-training 
programs and entry-level college courses. The JTPC 
study used a modified Delphi method or convergent 
consensus process (Conley, 2006) to identify, through 
content analysis of course materials, the prerequisite 
KSAs of job-training program courses in the same 
five occupational areas explored in the JSS study. 1 
The CCCA study also follows the convergent 
consensus approach to content analysis of course 
materials, but rather than five occupation areas, the 
CCCA study focuses on entry-level college courses, as 
represented by four course titles in mathematics and 


four course titles in courses with substantial reading 
demands. Course titles relevant to the NAEP 
mathematics framework included college algebra, 
finite mathematics, precalculus/calculus, and 
statistics. Titles relevant to the NAEP reading 
framework included English literature, psychology, 
U.S. government, and U.S. history. 


1 The five occupations were Automotive Master Technician; Computer Support 
Specialist; Heating, Ventilation, and Air Conditioning Technician; Licensed 
Practical Nurse; and Pharmacy Technician. 


CCCA STUDY OVERVIEW 


CCCA is a content analysis study with the primaty 
purpose of identifying the KSAs needed for a student 
to be prepared to enter an entry-level college course, 
without remediation. The methodology relies on a 
formal, systematic approach using expert judgment to 
measure and validate content and conclusions that are 
challenging to measure, yielding rich and useful 
information. Confidence in the findings is dependent 
on review of the reliability and validity associated with 
the pillars of the methodology: the institutional 
representativeness from which the study artifacts 
were drawn, the sufficiency of the artifacts to allow 
determinations of applicability, and the intra- and 
interrater reliability of the reviewers. 

Research Questions 

The CCCA study addresses four research questions 
for both mathematics and reading: 

1 . What are the prerequisite knowledge, skills, and 
abilities (hereafter referred to as prerequisite 
KSAs) in mathematics and reading to qualify 
for entry-level, credit-bearing courses that 
satisfy general education requirements? 

2. How do these prerequisite KSAs compare 
with the 2009 and 2013 NAEP mathematics 
and reading frameworks and item pools? 

3. How do these prerequisite KSAs compare 
with previous NAEP preparedness research; 
i.e., the descriptions of minimal academic 
preparedness requirements produced in the 
JSS research? 

4. How can these prerequisites inform future 
NAEP preparedness research; i.e., planning and 
analysis efforts relative to the 2013 12th grade 
NAEP mathematics and reading assessments? 


To achieve the goal of CCCA study comparability 
with the previous JSS study, the CCCA study relied 
on the same NAEP frameworks used in JSS, collected 
course materials and syllabi from the same (2009) 
academic year and used 2009 NAEP 12th grade items 
for comparison. During the study, 2013 items were 
available and were included in the study. 

CCCA Study Phases 

The CCCA study began in October of 2012 and 
spanned eighteen months. The first phase lasted nine 
months and consisted of identifying the institutional 
sample, identifying and collecting artifacts, and 
recmiting content reviewers and NAEP experts. The 
second phase lasted four months and encompassed 
reviewer training, coding, data summarization, and 
group meeting activities. The final phase lasted six 
months and included NAEP expert review of coding 
from the group meeting activities, data analysis, and 
report development. 

The three phases of the CCCA study are summarized 
below and illustrated in Figure 2: 

1. Establishing the foundation for the study, 
including creating a representative sample of 
institutions, collecting artifacts, organizing 
them into course packets, and selecting 
content reviewers and NAEP experts. 

2. Conducting content analysis activities 
including training and qualifying content 
reviewers, convening a series of coding events 
using the convergent consensus content 
review process, summarizing the data into 
detailed content maps of the KSAs, and 
completing a generalizability study (G-study). 





Figure 2. Three Phases of the CCCA Study 

Note. Artifact collection (EPIC staff); Convergent consensus (content reviewers); NAEP review (NAEP experts). 


3. Conducting NAEP expert reviews of the 

documentation, analyzing and synthesizing the 
results, and producing a final report addressing 
the four research questions of the CCCA study. 

Through the content analysis of artifacts (syllabi, 
assignments, assessments, and textbook excerpts 
assembled into course packets), the CCCA study 
identified the prerequisite mathematics- or reading- 
related knowledge, skills, and abilities (KSAs) 
necessary for students to be academically prepared, 
without remediation, for entry-level college courses. 
The result was a set of detailed content maps of the 
KSAs. A G-study was conducted to analyze variation 
and determine generalizability. 


The NAEP experts reviewed the content maps of 
prerequisite KSAs and developed narrative 
descriptions to summarize the prerequisite KSAs 
identified by the content reviewers. They then 
compared their narrative descriptions to the content in 
the items in the 2009 and 2013 12th grade NAEP item 
pools to analyze the correspondence between the 
CCCA KSAs and content assessed in the NAEP items. 
Next, the NAEP experts compared the narrative 
descriptions of the prerequisite CCCA KSAs to 
borderline performance descriptions and other content 
summaries captured in the JSS and JTPC studies. 


KEY TERMS 


The following section summarizes the key terms used 
in the CCCA study, developed in conjunction with 
the NAEP Glossary of Terms (U.S. Department of 
Education, National Center for Education Statistics, 
2013). The terms are organized in alphabetical order 
in the glossary in Appendix A. 

College Course Content Analysis (CCCA) — 

Acronym used throughout this study. 

KSA — Knowledge, skills, and abilities. 

Content Analysis — An objective and systematic 
technique for making replicable and valid conclusions 
about course content from course packets 
(Krippendorff, 2013). 

Institution — Postsecondary college or university that 
met representativeness criteria from which courses 
were selected and course artifacts were collected to 
form complete course packets. 

• Institution Inclusion Criteria — Criteria that 
defined the minimum requirements for an 
institution to become eligible to submit 
artifacts to be included in the CCCA study. 

Artifact — A syllabus, assessment, assignment, or 
textbook excerpt. Artifacts were systematically 
collected and organized into complete course packets. 

• Artifact Inclusion Criteria — Criteria, 
specific to a syllabus, assessment, assignment, 
or textbook excerpt, that defined the 
minimum requirement for the artifact to be 
included in a complete course packet. 


Extant Artifacts — Relevant artifacts from a 
repository of course materials that had been 
collected during previous studies by EPIC and 
that EPIC was given permission to use for 
additional studies. These artifacts from EPIC’s 
repository were included in the CCCA study. 

Course — A class offered at a postsecondary 
institution. The CCCA study focused on one of eight 
course titles with mathematics and reading content 
that fulfilled a set of course inclusion criteria (see the 
Course Packets section for the criteria). Artifacts were 
solicited for these courses and used to construct 
course packets. 

• Course Title — A grouping of courses (e.g., 
U.S. history) that would be expected to have 
similar content regardless of the institution at 
which a given course is taught. 

• Course Inclusion Criteria — Criteria that 
defined the minimum requirements for a 
course to be included in the study. 

• Course/ Artifact Submitter — A college-level 
instructor who submitted a minimum of three 
artifacts for a single course to be organized 
into a single course packet. 

• Online Submission Instrument — The 

platform where a Course/Artifact Submitter 
attests that the artifacts being submitted relate 
to a course that meets the Course Inclusion 
Criteria for the CCCA study and submits the 
required artifacts for a complete Course 
Packet. 




Course Packet — A set of three or more artifacts 
representing a single course title at one institution, 
including a syllabus, a textbook excerpt, and either an 
assignment or an assessment. There were a total of 28 
course packets for each course title used for four 
different purposes: training reviewers to participate in 
the study, qualifying reviewers to participate in the 
study, operational use by reviewers in Phase 2 of the 
study, and validation of results compiled from the 
reviewers. 

• Code/Coding — The category of a 
knowledge, skill or ability (KSA) statement, 
which is identified based on evidence within a 
course packet. The process of categorization 
conducted by the content reviewers is referred 
to as coding throughout the study. A coding 
scheme was developed with specific codes 
used by the content reviewers to facilitate 
analysis. 

• Coding Scheme — A systematic and formal 
structure for defining, identifying, and 
recording prerequisite knowledge, skills, and 
abilities in course artifacts during review. 

• Decision Rules — Guidance on how to 
identify evidence during the review process, 
applying the coding scheme to areas identified 
as potentially ambiguous. 

• Training Packet — One of two course 
packets in mathematics or one of two course 
packets in courses with substantial reading 
demands that were reviewed and coded for 
evidence of KSAs by the content reviewers 
during their training and prior to conducting 
qualifying reviews. 

• Qualifying Packet — One of two course 
packets in mathematics or one of two course 


packets in courses with substantial reading 
demands that were reviewed and coded by 
content reviewers after participating in 
training. The coding on these packets was 
used to assess content reviewers’ 
understanding of the review process. If 
adequate understanding was not demonstrated 
in the qualifying packet reviews, a content 
reviewer was released from participating 
further in the study. 

• Operational Packet — One of 80 course 
packets in mathematics or one of 80 course 
packets in courses with substantial reading 
demands, 20 per course title, reviewed and 
coded for evidence of KSAs by content 
reviewers. 

• Validation Packet — One of eight course 
packets in mathematics or one of eight course 
packets in courses with substantial reading 
demands, two per course title, reviewed by all 
content reviewers during the independent and 
group reviews to assess reviewer consistency. 
Reviewers’ codings on validation packets 
provided the data analyzed during the 
generalizability studies. 

NAEP Framework — The blueprint that guides the 
development of the NAEP assessment instrument 
and determines the content to be assessed by NAEP. 
For the purposes of the CCCA, a NAEP framework 
is a selection of content statements used as the basis 
for coding the evidence found in course packets. The 
2009 12th Grade NAEP Frameworks for Reading 
and Mathematics were used, and neither framework 
has changed since 2009. 

• Subject Area — One of the areas assessed (or 
planned for assessment) by NAEP, which 
includes but is not limited to the arts, civics. 


economics, foreign language, geography, 
mathematics, reading, science, U.S. history, 
world history, and writing. CCCA examined 
two subject areas: mathematics and reading. 
The term Course Type has been used in 
CCCA-related materials and is considered 
synonymous with the term Subject Area. 

Objective — Level of statement within the 
NAEP mathematics and reading framework 
referred to for content review and analysis. 

Subtopic — A level in the hierarchy of an 
assessment framework; a statement that is 
more general than an Objective in the NAEP 
mathematics framework. For example, the 
NAEP mathematics framework is organized 
from highest level to lowest level as follows: 
Subject Area- 1. Content Area- 1.1 Subtopic- 
1.1. a Objective. For example, Mathematics- 
1. Number and Properties- 1.1 Number 
Sense- 1.1. a Represent, interpret, or compare 
expressions for real numbers, including 
expressions using exponents and logarithms. 

Standard — A level in the hierarchy of an 
assessment framework; a statement that is 
more general than an Objective in the NAEP 
reading framework. For example, the NAEP 
reading framework is organized from highest 
level to lowest level as follows: Subject Area- 
1. Cognitive Domain: 1.1 Standard: 1.1. a 
Objective. For example, Reading- 1. 
Locate/Recall- 1.1 Locate or recall textually 
explicit information and make simple 
inferences within and across both literary and 
informational texts- 1.1. a Locate or recall 
specific information such as definitions, facts, 
and supporting details in text or graphics. 


• NAEP Item and NAEP Item Pool — A set 

of basic scorable parts of the NAEP 
assessment or test questions (NAEP Items) 
available for administration. NAEP items are 
either multiple-choice or constructed- 
response. The CCCA study included a 
comparison of prerequisite KSAs with the 
2009 and 2013 12th Grade NAEP item pools 
in mathematics and reading. 

• Holistic Review Statement — A statement 
of knowledge, skills and abilities identified 
during the Holistic Review, which is 
conducted before reviewers are asked to use 
the NAEP Frameworks as a reference. 
Holistic Review Statements are outputs of 
a training exercise that familiarizes the 
Content Reviewers with all of Course 
Packets they will review in the study, while 
also allowing reviewers to suggest prerequisite 
KSAs in their own words, based on the 
evidence identified in the Course Packets. 

• NAEP-Specific KSA — A statement of 
knowledge, skills, and abilities that is 
articulated in the NAEP framework. The 
NAEP-specific KSAs may not be evident as 
prerequisite in the course packets. 

• NAEP-Specific KSA Exclusion — A 

selection of text from a NAEP-specific KSA 
that does not apply to the coding by a content 
reviewer or content reviewer group. 

• Non-NAEP Additional KSA — A statement 
of knowledge, skills, and abilities that is not 
listed in the NAEP framework. These include 
the Holistic Review Statements that 
content reviewers within each Subject Area 
deemed appropriate to integrate into the 
operational Coding process. 


Reviews — Identification and coding of evidence of 
knowledge, skills, and abilities within the content of 
course artifacts compiled into packets by experts. 

• NAEP Expert — A highly qualified doctoral- 
level expert in mathematics or reading content 
who served as an advisor to the CCCA study. 
Each NAEP expert had specialized expertise 
on the NAEP, including a deep understanding 
of the NAEP and participated in the 
development of the NA E P framework. All 
had extensive experience with previous 
NAEP studies, including the JSS and JTPC 
studies. 

• Content Reviewer — A highly qualified and 
trained college-level mathematics or reading 
content expert who reviewed course packets 
for evidence of KSAs for the CCCA study. 

• Holistic Review — An initial general review 
of course packets conducted as a training 
exercise prior to content reviewer training on 
the NAEP frameworks and the convergent 
consensus review methodology. 

• Content Review — The application of the 
coding scheme to a course packet, an 
assembly of course artifacts. Content reviews 
were conducted independently and in a group. 


• Content Review Group — A group of three 
content reviewers who independently and 
collectively reviewed the same set of course 
packets. 

• Independent Review — A process, preceding 
group review, where content reviewers 
reviewed a set of 28 course packets 
independendy as part of a convergent 
consensus process. 

• Group Review — A process, following 
independent review, where content reviewers 
attended a facilitated meeting to review the 
same 28 course packets and resolve coding 
disagreements as part of a convergent 
consensus process. 

• Content Maps — Visual representations of 
the group review applicability ratings and 
NAEP-Specific KSA Exclusions related to 
each of the NAEP-Specific KSAs. 

Applicability ratings represent the degree to 
which a KSA relates to evidence in a course 
packet. 

Generalizability Study — A statistical analysis of the 
reliability of using content reviewers to code course 
packets and the results of CCCA study under specific 
constraints. This should not be confused with the 
term “generalizability” that refers to the extension of 
these research findings and conclusions to the 
population at large. 


STUDY METHODS 


This study relied on a convergent consensus process 
to determine the mathematics and reading KSAs that 
are prerequisite to entry-level college courses. Dr. 
David Conley developed the convergent consensus 
model by drawing upon lessons learned from the 
RAND Corporation’s Delphi method (Dalkey, 1969) 
and incorporating principles of evidence-centered 
design, an evidentiary reasoning approach applied in 
educational assessment development (Cooke, 1991). 
The convergent consensus process systematically 
solicits judgments from individuals with specific, 
relevant experience and expertise by developing and 
employing an explicit decision-making criterion that 
informs the judgments that experts make throughout 
the process. The goal at each stage is to draw upon 
the experts’ perspectives of the subject area in 
question and to activate this expert knowledge within 
an explicit decision framework, grounding each 
decision in specific evidence drawn from course 
artifacts. 

Although high levels of expert agreement often occur, 
they are not expected initially. The level of agreement 
increases as content reviewers engage in discussions 
of specific, contextualized differences. Consensus is 
the goal but not a requirement. In the final analysis, 
experts may not reach agreement on every item, and 
facilitators do not force consensus when experts have 
substantial or deep-seated differences on a particular 
code. Percentages of agreement are calculated as a 
finding of the study. 

EPIC implemented a strong set of project 
management and research processes to ensure reliable 
and valid data collection, management, and analysis, 
as well as comprehensive and accurate reporting of 


study findings. The next sections detail the data 
sources, procedures, and results relative to each of the 
three phases of the study. 

Phase I: Establishing the Foundational 
Study Components 

The quality of the results of the CCCA study is based 
on the quality of its three key components: 

1. Institutional representativeness 

2. Course packet sufficiency 

3. Selection of content reviewers and NAEP 
experts 

First, institutional representativeness was defined for 
the CCCA study. Next, the criteria for a complete 
course packet were developed and artifacts were 
collected to compose course packets from institutions 
that would contribute to a representative sample. 
Finally, profiles of appropriate content experts were 
developed for the CCCA study and reviewers were 
recmited to match these profiles. 

Institution Sample Selection and Recruitment 
Research staff defined the population of institutions 
using the 201 1 Integrated Postsecondary Education 
Data System (U.S. Department of Education, 

National Center for Education Statistics, 201 1). 
IPEDS is a primary source for data on colleges, 
universities, and technical and vocational 
postsecondary institutions in the U.S. 

From the 7,643 postsecondary institutions identified, 
the team established sampling parameters for the 
study. Programs shorter than two years, proprietary 
institutions, and those with a special focus are not 




Table I. Institutional Criteria and Category Definitions 


Criterion 

Category 

Program type: Institutions are classified based on length of 
program. Institutions with programs that are shorter than 2 

2-year: Programs are longer than 2 but shorter than 4 years 

years are excluded from the CCCA and national population 
datasets. 

4-year: Programs are 4 years or longer 

Enrollment size: Institution size is based on the total 
number of students enrolled for credit in Fall 2011. 

Small: Enrollment of 4,999 or fewer 

Medium: Enrollment between 5,000 and 9,999 


Large: Enrollment of 10,000 or more 

Geographic region: Institutions are classified by region (East 
or West) based on which state they are located in. 

East: AL, AR, CT, DC, DE, FL, GA, KY, LA, MA, MD, ME, 

MS, NC, NH, NJ, NY, PA, Rl, SC, TN, VA, VT, and WV 

West: AK, AZ, CA, CO, HI, IA, ID, IL, IN, Ml, OH, Wl, KS, 
MN, MO, MT, ND, NE, NM, NV, OK, OR, SD, TX, UT, WA, 
and WY 

Control of institution: Institutions are identified by two 
classifications: public or private. Institutions that are private 
for-profit are excluded from the dataset. 

Public: An educational institution in which programs and 
activities are operated by publicly elected or appointed school 
officials and that is supported primarily by public funds. 

Private: A private institution in which the individual(s) or 
agency in control receives no compensation other than 
wages, rent, or other expenses for the assumption of risk. 
These include independent, not-for-profit schools and those 
affiliated with a religious organization. 


included in the sampling frame of institutions for this 
study due to expected differences in course content, 
rigor, and requirements. 2 In addition, institutions with 
missing values on four institutional criteria and those 
with values that fell outside the categories listed below 
(e.g., institutions not within the 50 U.S. states, such as 
Puerto Rico) also were excluded because they 
introduced unknown variance and/ or cultural and 
language variance. After excluding these institutions, 
the researchers identified a population of 3,010 
institutions, from which they identified a probability 
sample of 184 institutions that would yield 184 course 
packets for analysis, assuming that each institution 
contributed one course packet. The sample was 
stratified by program type, size of enrollment, 


2 Special focus schools include doctoral research universities; master’s colleges 
and universities; medical schools; other health professions schools; schools of 
engineering; other technology-related schools; schools of business and 
management; schools of art, music, and design; schools of law; and other special 
focus institutions. 


geographic region, and institutional control (see Table 
1 for detail). 

Ultimately, 169 unique institutions submitted course 
artifacts. Table 2 describes the characteristics of the 
169 institutions and summarizes the artifact data 
collection process related to representativeness. 
Course packets drawn from these 169 institutions 
were used for training and qualifying reviewers, 
validating coding consistency (validation packets), and 
content analysis (operational packets). In addition, 
course artifacts were collected from previous EPIC 
studies to augment the artifacts collected solely for 
the CCCA study. 

Table 3 presents the number of institutions, number 
of packets, and distribution of the packets as required 
for the purpose of creating training, qualifying, 
operational, and validation packets. 




Table 2. Percentages of Institutions, by Characteristic, in the Population and in the Sample of 169 Institutions Submitting 
1 84 Course Packets 


Institutional 

characteristic 

Category 

Population % 

Mathematics 
sample % 

Reading sample % 


2-year 

35 

30 

24 

Program type 


4-year 

65 

70 

76 


Small 

64 

51 

50 

Enrollment size 

Medium 

17 

20 

22 


Large 

19 

29 

28 


East 

50 

45 

41 

Geographic region 


West 

50 

55 

59 


Public 

53 

61 

58 

Control of institution 


Private 

47 

39 

42 


Course Packets 

After the characteristics of the representative 
institution sample were determined, course packets 
were compiled using submitted artifacts. Complete 
course packets included at least three artifacts: a 
syllabus, one or more assessments or assignments, 
and an excerpt from a primary textbook. Syllabi were 
intended to provide evidence of course goals and 
objectives. Assignments and assessments were 
intended to identify what students are expected to 
demonstrate as evidence of what they have learned. 
The text excerpt was intended to identify a relevant 
reference available to the student. Some course 
packets included supplemental assignments, 
assessments, and other artifacts. 

Course packets were solicited from colleges across the 
U.S. to serve as the basis of the content review for 
prerequisite KSAs to entry-level college courses. The 
research team first established a set of course-level 
inclusion criteria appropriate to identify prerequisites 
for entry-level college courses. Courses were included 
if they met the following criteria: 


• were credit bearing 

• were generally understood to be taken 
frequently by entry-level students 

• had no college -level prerequisites 3,4 

• fulfilled general education requirements 3 4 5 

• were entry level 

• were not remedial 

• were not honors level 

• represented actual courses taught in academic 
year 2009-2010 or 2010-2011 

• were not identified with a specific major 

• contained syllabi with sufficient information 
about what was taught in the course (This 
information would be used to identify 
relevant mathematics and reading KSAs.) 


3 Courses with corequisites or noncourse prerequisites were coded as having “no 
college-level prerequisites.” Examples of noncourse prerequisites include but are 
not limited to sufficient scores on placement exams and minimal skills in the 
subject area (e.g., “minimal algebra skills”). 

4 For courses collected and coded as a single course but taught over multiple 
terms as a sequence (e.g., 101 taught fall term, 102 taught winter term, and 103 
taught spring term), only the first course in the sequence was included in the 
sample. The subsequent courses were excluded from the CCCA study under the 
assumption that completion of the first course in the sequence was required to 
advance to the second course. 

5 Policies indicating that credits earned in programs offering associate’s degrees 
are transferrable to programs offering bachelor’s degrees are dictated by 
articulation agreements between institutions and state departments of education. 
Representatives from the two-year institutions submitting courses for inclusion 
in the study verified that the submitted course fulfills general education 
requirements toward a four-year degree. 




Table 3. Institutions Sampled and Course Packets Required for the CCCA Study 


Process 

Institutions 

Course packets 
required 

The initial assumption was that 184 institutions drawn from a stratified 
probability sample would contribute 1 course packet each. 

184 

184 

The researchers reviewed the extant artifact base, found a number of 
courses that fit the institution sample requirements and included them in 
the total CCCA artifact collection. Course packets were selected from 
representative institutions on a rolling basis. For example, once a 
sufficient number of 2-year, large public institutions from the West had 
provided course packets, no more artifacts were collected from that 
type of institution. 

• 155 institutions submitted 1 course packet = 155 packets 

• 13 institutions submitted 2 course packets = 26 packets 

• 1 institution submitted 3 course packets = 3 packets 

169 

184 

Four course packets were subtracted from the total and used for 
training content reviewers: 2 from mathematics courses and 2 from 
courses with substantial reading demands. 

4 

4 

Four course packets were subtracted from the total and used for 
qualifying content reviewers: 2 from mathematics courses and 2 from 
courses with substantial reading demands. 

4 

4 

Sixteen course packets were subtracted from the total and used for 
validation: 2 from each of the eight course titles. 

16 

16 

After the training, qualifying and validation packets were subtracted, 1 5 1 
institutions contributed 160 operational packets — 20 packets for each of 
the eight course titles — providing the set of operational packets for 
content analysis. Seventy-seven institutions provided packets for courses 
with substantial reading demands, and 74 institutions provided packets 
for mathematics courses. 

151 

160 

One precalculus/calculus course and one institution were removed 
during the NAEP mathematics expert review. The NAEP experts who 
convened after the group content reviewers strongly suggested 
removing a multivariable calculus course because it did not represent 
entry-level mathematics content, although it had met the inclusion 
criteria of being entry level at the institution that submitted it. It was not 
replaced due to the timing of the ineligibility determination. 

150 

159 


Course packet recruitment began with the research 
team examining extant artifacts collected from 
previous EPIC studies to identify those that satisfied 
the CCCA institution sample and course inclusion 
criteria. An inventory of extant course artifacts from 
prior EPIC projects identified 52 mathematics 
courses and 382 courses with substantial reading 
demands for potential inclusion in the CCCA study. 
Some of these extant artifacts were composed of 
complete course packets and others consisted of 


course packets that were partially complete (i.e., 
missing one artifact type), given the requirements of 
the CCCA study. For courses with partially complete 
sets of artifacts, staff sent emails to the original faculty 
contacts requesting that they submit any missing 
artifacts. Those who completed their course packets 
were given an honorarium. Of the 184 complete 
course packets used in the study, 81, or 44%, were 
from previous EPIC studies. These 81 extant course 
packets include 5 mathematics and 55 reading 




operational course packets, all 4 training course 
packets, 3 of 4 qualifying packets, and 14 of 16 
validation packets. 

The research team conducted a gap analysis analyzing 
the difference between the characteristics of the 
institutions from which the extant course artifacts to 
be used as operational packets were drawn and the 
targeted number of institutions within the sampling 
frame to determine recmitment parameters for 
collecting the remaining 103 course packets. 

The research team contacted postsecondary 
instructors from the targeted institutions to request 
courses artifacts. Instructors submitted artifacts 
through an online submission form. To verify that 
courses from selected institutions met the inclusion 
criteria, the research team conducted online reviews 
of course catalogs and department websites. An initial 
recmitment email sent to a selected course instructor 
or a department contact included an overview of the 
CCCA project, a request for participation via an 
online submission process, mention of an 
honorarium, the date window for artifact submission, 
and an opportunity to opt out of participation in the 
CCCA study. 

Before submitting artifacts, instructors had to verify, 
through an online survey, that the course satisfied the 
course inclusion criteria. The instructor or department 
contact was considered unresponsive after the 
research team attempted one follow-up phone call 
and one follow-up email request. This process was 
followed until the necessary number of course 
artifacts was collected. To obtain the necessary 
number of artifacts to create sufficient packets, all 
institutions on the list were contacted. Of all the 
course titles, it was most challenging to find calculus 
courses that had no prerequisites, as most calculus 
courses require college algebra as a prerequisite. See 
the artifact solicitation email in Appendix B. 


Institutions were allowed to submit artifacts for more 
than one course if the courses represented different 
course titles. For courses with substantial reading 
demands, six institutions provided artifacts for two 
courses. In mathematics, five institutions provided 
two courses. Three institutions provided artifacts for 
courses in both subject areas. It was never the case 
that an institution submitted artifacts for more than a 
single course within a course title. 

The research team sent 1,570 emails and attempted 
633 follow-up phone calls to instructors or 
departmental contacts. New course packets submitted 
using the online submission instrument represented 
67 of 184 (36%) complete course packets required for 
the study. The effort to recruit new course artifacts 
for packets yielded 43 mathematics operational course 
packets and 21 operational course packets for courses 
with substantial reading demands. In addition, the 
effort yielded one of four qualifying and two of 16 
validation packets. Using extant packets and newly 
recmited packets, the research team collected 148 of 
the 184 total required packets, 124 of which were 
operational. 

To collect the remaining packets that were needed, 
the research team reviewed websites using general 
keyword Google searches. For example, the Google 
search “college algebra syllabus 2010 community” 
produced online syllabi for college algebra courses 
taught in community colleges during the 2010 
academic year. 

Mining online websites yielded the remaining 36 
complete course packets required for the study. These 
comprised 20% of the 184 total course packets. 
Thirty-two mathematics operational course packets 
and four operational packets for courses with 
substantial reading demands were created from course 
materials mined from websites. The combined effort 
produced a total of 184 complete course packets, 160 


of which initially were operational packets. Near the 
end of the study, after the content reviews were 
completed, one multivariable calculus packet was 
eliminated, leaving 159 operational packets. The 
NAEP experts, convened after the group content 
reviews, recommended removing a multivariable 
calculus course from the study because it did not 
represent entry-level mathematics content, even 
though it had met the inclusion criteria of being entry 
level at the institution that submitted it. It was not 
replaced due to timing of the ineligibility 
determination. 

Table 4 summarizes the process and outcomes 
described in this section. 

Course Title Selection 

The college courses analyzed in this study represent the 
most frequently taken entry-level, credit-bearing 
mathematics courses or courses that require college- 
level reading, which satisfy general college graduation 
requirements. To identify frequently taken course titles 


that were potential candidates for this study, the 
research team referred to a report prepared by the 
National Center for Education Statistics (NCES) that 
identified the top 30 most studied postsecondary 
courses (U.S. Department of Education, National 
Center for Education Statistics, 2004). The research 
team also drew on previous research to narrow the list 
to eight course titles that were likely to meet the course 
inclusion criteria and contain sufficient mathematics 
and reading content to permit reviewers to determine 
the KSAs associated with minimal academic 
preparedness without need for remediation (Conley, 
2011). The course titles selected for the CCCA study 
for mathematics are precalculus/ calculus, college 
algebra, finite mathematics, and statistics. The course 
titles selected for courses with substantial reading 
demands are English literature, psychology, U.S. 
government, and U.S. history. Appendix C provides 
the names of the courses associated with each course 
packet analyzed in the study. 


Table 4. Course Packet Recruitment Processes and Outcomes 


Course packet type 

Extant artifact 

database 

Online submission 

instrument 

Internet search 

Total 

Mathematics total 

14 

46 

32 

92 

Training 

2 

0 

0 

2 

Qualifying 

1 

1 

0 

2 

Validation 

6 

2 

0 

8 

Operational 

5 

43 

32 

80 

Reading total 

67 

21 

4 

92 

Training 

2 

0 

0 

2 

Qualifying 

2 

0 

0 

2 

Validation 

8 

0 

0 

8 

Operational 

55 

21 

4 

80 

Course packet total 

81 

67 

36 

184 


Note. The count of course packets from the extant courses includes complete and partially complete course packets. Table 
includes artifacts collected for a mathematics course packet that was disqualified from the study after the content review. 




Number of courses in each course title. Because 
this study involved a content analysis of course 
documents, the criteria for determining adequate 
sample size of course packets were derived from the 
principle of redundancy. In other words, the sample 
would be considered to be sufficient at the point 
where new packets were yielding information about 
prerequisites that was essentially redundant with that 
gathered from all previous packets. Results from 
previous studies conducted by EPIC, including the 
JTPC study, and from generally accepted practices in 
this field of research (Krippendorff, 2013), indicate 
that a sample of 20 courses per course title is 
sufficient to achieve redundancy at a level that 
supports valid and generalizable inferences. The 
CCCA study used this sample size of 20 courses per 
course title to achieve sufficient redundancy. 

Artifact Inclusion Criteria 

Artifacts that satisfied both the institution and course 
inclusion criteria also had to satisfy a set of artifact 
inclusion criteria before being compiled into a course 
packet. 

To be considered complete, a course packet needed 
three artifacts: a syllabus, an assignment or 
assessment, and an excerpt from a primary textbook. 
Course packets were not restricted to just three 
artifacts, so some also contained supplemental 
assignments, assessments, and other artifacts. The 
most common supplemental artifacts were study 
guides, grading mbrics, and assignment lists. 

Inclusion criteria for each of the three required 
artifacts are described below. 

Syllabus. Based on earlier studies (Conley, 2008), the 
research team expected that syllabi from some college 
courses would not provide sufficient information or 
detail for content analysis. Because the artifacts 
needed to provide a level of data richness that would 


be sufficient for reviewers to identify evidence of 
mathematics or reading KSAs taught in the course, 
each syllabus was reviewed for sufficient mathematics 
or reading information to conduct content analysis. 

Information-lacking syllabi generally provide a brief 
course description, some course policies, and nothing 
more. Sufficient syllabi generally include detailed 
course descriptions, policies, schedules, objectives or 
goals, and brief descriptions of all course activities, 
assignments, and assessments. Sufficient syllabi may 
also describe the learning progression and allow for a 
more accurate identification of the prerequisite KSAs 
necessary for entry into the course. 

Textbooks. An earlier study (WestEd & EPIC, 2013) 
indicated that primary textbooks provide a wealth of 
information about the KSAs required in a course. 

Past content reviewers have identified textbooks as 
the most helpful source of evidence for identifying 
prerequisite KSAs. The course text artifact was used 
heavily as a source of evidence for identifying KSAs 
across all the occupational areas reviewed in the JTPC 
study for both mathematics and reading. 

The textbook collection process identified and 
obtained textbook information from instmctors using 
three methods to obtain textbook excerpts: an online 
data mining process, borrowing textbooks, and 
purcha sing/ renting textbooks. The research team 
applied a hierarchical decision rule framework for 
identifying primary textbooks and selected textbook 
excerpts for each course packet. 

The decision rule frameworks typically applied to 
precalculus/calculus, college algebra, statistics, finite 
mathematics, psychology, U.S. government, and U.S. 
history textbooks. For courses included under the 
English literature course title that use novels rather 
than textbooks as required texts, the first chapter(s) of 
the first novel to be covered in the class was included. 


T extbook Selection Decision Rules 

1 . For courses with only one textbook, that 
textbook was considered the primary textbook. 

2. For courses with two or more required 
textbooks and the syllabus did not indicate a 
primary textbook, the following guidelines were 
used: 

a. If the institution was contacted through the 
data collection process, the institution was 
asked to identify the primary textbook. 

b. If a course schedule was available, the 
textbook covered earliest in the course 
schedule was considered the primary 
textbook. 

c. If no course schedule was available, the 
following rules applied to gather the most 
appropriate textbook-based artifact: 

i. if one of the textbooks was used in 
multiple course titles, it was selected; 

ii. if topics were listed in the syllabus, the 
research team inferred the primary 
textbook from the coverage of early 
topics; and 

iii. if no other information was available, 
the textbook listed first was selected. 

3. If no textbook was identified, the institution was 
contacted through the supplemental artifact 
collection process and requested to submit a 
primary textbook or provide a textbook-based 
artifact for consideration. 


See Appendix D for the complete description of 
artifact collection inclusion criteria and decision rules 
applied to this study. 

Assessments and assignments. At least one 
assessment or assignment was required in every 
complete course packet. Because the goal of this study 
was to identify prerequisite KSAs, only assessments or 
assignments provided early in the course schedule were 
acceptable for inclusion. Midterm exams were accepted 
if there were no assignments or assessments available 
from earlier in the course. Because the artifacts needed 
to provide a level of data richness that would be 
sufficient for reviewers to identify evidence of 
mathematics or reading KSAs taught in the course. 


Textbook Excerpting Decision Rules 

1 . The table of contents was included when 
present. 

2. One or two chapters were included depending 
on the length of the chapters. Typically, 
textbook artifacts did not exceed 100 pages. 

a. If the course schedule was available, the 
first chapter(s) covered in the course was 
excerpted. 

b. If no course schedule was available, the first 
chapter(s) in the textbook was excerpted. 


each assignment and assessment was reviewed for 
sufficient mathematics or reading information to 
conduct content analysis. 

Many mathematics course assignments or assessments 
consisted of a list of problems directly from the 
course textbook. When available, the research team 
collected supplementary assessments or assignments 
from these courses. In cases where the only 
assessments or assignments were from the textbook, 
the problem set for the first assessment or assignment 
was included in the textbook excerpt. 

Complete Course Packets 
Table 5 displays counts of artifacts within all the 
course packets by type, subject area, and course tide. 
Within course tides with substantial reading demands, 
U.S. history had the most artifacts with 98, and 
English literature had the fewest with 83. Course 
packets from courses with substantial reading 
demands relied the heaviest on assignments as a 
supplement to syllabi and text excerpts, with a total of 
110 assignment artifacts across course packets. 

Within mathematics courses, finite mathematics 
course packets had the fewest course artifacts with 74 
artifacts, while precalculus/ calculus had the most 
mathematics artifacts with 78. Mathematics relied 
more heavily on assessments as a supplement to 


syllabi and text excerpts, with 60 assessment artifacts 
across the packets. 

Mathematics and courses with substantial reading 
demands had similar counts of “other” artifacts (i.e., 
study guides, grading rubrics, assignment lists, 
classroom handouts). All assessments for reading are 
either quizzes or exams produced by the instructor of 
the course and are not from the textbook. 

Across the course packets from courses with 
substantial reading demands, the U.S. history course 
title was unique in that there were more assessments 
than assignments; the opposite was true in 
psychology, U.S. government, and English literature. 
English literature relied solely on assignments with 
one exception. Overall, each mathematics course tide 
had approximately the same number of artifacts. 
College algebra relied heavily on assignments, whereas 
finite mathematics, precalculus/calculus, and statistics 
all relied more heavily on assessments. 


In addition to the artifacts, each course packet 
contained a title page that displayed the course title 
(e.g., U.S. history), course name (e.g., American History 
Since 1877), course number (e.g., HIST 122), a list of 
artifacts included in the course packet, the primary 
textbook title and author, how the textbook excerpt 
was identified, and what was included from the 
textbook. All information that could identify the 
institution that submitted the course was redacted 
from the course packet. Appendix E contains a sample 
mathematics course packet and a sample packet from a 
course with substantial reading demands. 

NAEP Grade 12 Reading and Mathematics 
Frameworks 

This study used versions of the 2009 Grade 12 NAEP 
mathematics and reading frameworks modified for 
alignment studies (U.S. Department of Education, 
National Center for Education Statistics, 2009; U.S. 
Department of Education, 2009). Appendix F 
contains the 2009 Grade 12 NAEP frameworks. 


Table 5. Number and Type of Artifacts in all Course Packets by Subject Area and Course Title 


Course title 

Syllabus 

Assessment 

Assignment 

Textbook 

Other 

Total 

Mathematics total 

92 

60 

44 

92 

16 

304 

Precalculus/calculus 

23 

17 

12 

23 

3 

78 

College algebra 

23 

1 1 

16 

23 

4 

77 

Finite mathematics 

23 

13 

8 

23 

7 

74 

Statistics 

23 

19 

8 

23 

2 

75 

Reading total 

92 

40 

1 10 

94 

17 

353 

English literature 

23 

1 

32 

23 

4 

83 

Psychology 

23 

8 

32 

23 

1 

87 

U.S. government 

23 

1 1 

26 

24 

1 

85 

U.S. history 

23 

20 

20 

24 

1 1 

98 

Total artifacts 

184 

100 

154 

186 

33 

657 


Note. Table includes artifacts collected for all course packet types, including training, qualifying, validation and operational 
course packets. Table includes artifacts collected for a mathematics course packet that was disqualified from the study after 
the content review. 




The 2009 NAEP Grade 12 reading framework is 
structured as follows: 

1. Cognitive domain 

1.1. Standard 

1.1. a Objective 

For example, as shown below, the second reading 
cognitive domain is “2. Integra te/Interpret: Make 
complex inferences within and across texts,” a standard 
within this domain is “2.1. Integrate/ Interpret: Make 
complex inferences within and across both literary and 
informational texts,” and an objective within that 
standard is “2.1. a. Describe problem and solution, or 
cause and effect.” Table 6 outlines the three cognitive 
domains and 10 standards in the reading framework, 
and identifies the number of objectives, 37. 


Table 6. 2009 NAEP Grade 12 Reading Framework 


Cognitive domain: Standard (number of objectives) 


Locate/recall 

• Locate or recall textually explicit information and 
make simple inferences within and across both literary 
and informational texts (I) 

• Locate or recall textually explicit information and make 
simple inferences within and across literary texts (5) 

• Locate or recall textually explicit information and 
make simple inferences within and across 
informational texts (4) 

I ntegrate/i nterpret 

• Make complex inferences within and across both 
literary and informational texts (6) 

• Make complex inferences within and across literary 
texts (5) 

• Make complex inferences within and across 
informational texts (5) 

• Apply understanding of vocabulary to comprehension 
of both literary and informational texts (I) 

Critique/evaluate 

• Consider both literary and informational texts 
critically (3) 

• Consider literary texts critically (3) 

• Consider informational text critically (4) 


For mathematics, the framework is structured as 
follows: 

1. Content Area 

1.1. Subtopic 

1.1. a. Objective 

Objectives are the most specific level in the 
organizational structure of the NAEP frameworks. For 
example, as shown below, the first mathematics subject 
area is “1. Number properties and operations,” a 
subtopic within that domain is “1.1. Number sense,” 
and an objective within that subtopic is “1.1. a. 
Represent, interpret, or compare expressions for real 
numbers, including expressions using exponents and 
logarithms.” Table 7 outlines the five content areas and 
24 subtopics in the mathematics framework, and 
identifies the number of objectives, 130. 

For the purposes of the CCCA study, the levels 
within the organizational structure of the NAEP 
frameworks were operationalized differently for 
coding mathematics and reading. The reading 
objective-level statements must be read within the 
context of the standard and cognitive domain levels. 
That is, content reviewers were required to interpret 
the objective in the context of its standard and 
domain. If any part of the “domain + standard + 
objective” was deemed not prerequisite but other 
parts were, then the parts that were not were 
identified as KSA exclusions. In mathematics, the 
hierarchy differs, and the content areas and subtopics 
are organizers for the objectives, rather than context 
for interpretation. As such, only the objective-level 
statements were considered for identifying KSA 
exclusions. 

Participants 

The research team recruited two separate groups of 
reviewers for the CCCA study: content reviewers and 
NAEP experts. 




Table 7. 2009 NAEP Grade 12 Mathematics Framework 


Content area: Subtopic (number of objectives) 


Number properties and operations: 

• Number sense (4) 

• Estimation (3) 

• Number operations (5) 

• Ratios and proportional reasoning (2) 

• Properties of number and operations (4) 

• Mathematical reasoning using numbers (2) 

Measurement: 

• Measuring physical attributes (6) 

• Systems of measurement (5) 

• Measurement in triangles (7) 

Geometry: 

• Dimension and shape (4) 

• T ransformation of shapes and preservation of properties (6) 

• Relationships between geometric figures (7) 

• Position, direction, and coordinate geometry (8) 

• Mathematical reasoning in geometry (5) 

Data analysis, statistics, and probability: 

• Data representation (6) 

• Characteristics of data sets (7) 

• Experiments and samples (5) 

• Probability (9) 

• Mathematical reasoning with data (5) 

Algebra: 

• Patterns, relations, and functions (7) 

• Algebraic representations (7) 

• Variables, expressions, and operations (7) 

• Equations and inequalities (6) 

• Mathematical reasoning in algebra (3) 


Content reviewers. The research team recruited and 
selected content reviewers who had relevant content 
expertise. Mathematics reviewers were informed and 
knowledgeable about the mathematics requirements for 
placement, without remediation, into entry-level, credit- 
bearing courses in mathematics. Reading reviewers were 
informed and knowledgeable about the reading 
requirements for placement, without remediation, in 
introductory courses that require extensive reading and 
fulfill general education requirements. 


Individuals with the following types of experience 
were recmited: 

• Instructors of two- and four-year 
postsecondary, entry-level, credit-bearing 
mathematics or English and social science 
courses that fulfill general education 
requirements for a four-year degree program 

• Instructors of remedial or developmental 
mathematics or courses with substantial 
reading requirements in postsecondary 
institutions 

• Instructors of postsecondary mathematics or 
English and language arts who have 
participated directly in the development of 
entry-level placement tests for a 
postsecondary institution 

The mathematics reviewers must have taught 
postsecondary courses fulfilling general education 
course requirements in mathematics. Reading 
reviewers must have specialized in reading pedagogy 
or have taught postsecondary courses in literature or 
one of the social sciences or humanities with 
extensive reading demands, and must be informed 
and knowledgeable about the reading requirements 
for course placement without remediation. 

The pool of potential content reviewers was initially 
reduced through a curriculum vitae rating process that 
ensured only instmctors that fit the criteria above were 
invited to participate in the study. The research team 
provided the Governing Board with a list of the 
individuals meeting the selection criteria for approval 
prior to recruitment. Content reviewers with extensive 
qualifying credentials, experience participating in the 
JSS or JTPC studies, or those from a pool of trained 
EPIC mathematics and reading content experts who 
had substantial experience with convergent consensus 
artifact review and content analysis were given priority 
for selection. The recruitment process resulted in a 




group of 32 reviewers (16 in mathematics and 16 in 
reading) divided into eight content review teams, each 
composed of three content reviewers and one 
alternate. Reviewers were paid for participation in the 
study. 

NAEP experts. A panel of NAEP experts, three in 
mathematics and three in reading, provided guidance 
during each phase of the CCCA study. All the NAEP 
experts provided advisory support during the 
planning phases of the study. Four of the six NAEP 
experts attended the group review meetings as 
advisors. All the NAEP experts convened after the 
content review to guide the interpretation of findings 
related to this study’s four research questions. 

Individuals serving as NAEP experts possess 
extensive knowledge of both the NAEP framework 
and the mathematics or reading content domains 
generally. The research team successfully recruited 
individuals who served on NAEP expert panels 
during the JSS and JTPC studies. The final list of 
NAEP experts was presented to and approved by the 
Governing Board. 

Phase 2. Conducting the Content 
Analysis 

The CCCA study employs a convergent consensus 
approach. This method relies on multiple criterion- 
based expert analyses designed to reach consensus on 
the relationship between the elements being judged 
and the criteria used to judge them. Expert reviewers 
must agree that the elements being judged meet the 
judging criteria. This method is different from more 
common methods where individual raters work in 
isolation to rate an item or element and then the 
ratings are compared to generate a traditional 
reliability statistic. This more common approach 
works well when the elements being rated are of a 
small grain size, such as a test item or an individual 
standard statement, for example, and the criteria to 


rate them can be applied in a reasonably 
straightforward or mechanical fashion. For example, 
does the item cover equations or functions? Or, does 
the standard require analytic thinking or only 
procedural thinking? Reaching judgments of this type 
can be achieved with minimal training with little to no 
oversight during the rating process. 

Flowever, analyzing artifacts that are more complex 
and at a much higher aggregation level than a test 
item requires a different approach. It begins with the 
notion that experts by their very nature are likely to 
interpret complex documents somewhat differently 
and that rating in isolation in fact obscures the deep 
understanding that experts bring to the analysis and 
rating process. Convergent consensus begins with 
individual ratings but differs from more commonly 
employed methods by requiring reviewers to identify 
the evidence or data source that supports each 
individual judgment. In this sense, convergent 
consensus incorporates and is built upon an evidence- 
centered design approach more than a classical rater 
reliability model (Mislevy, 2005). Next, reviewers 
share their judgments with other reviewers. Areas of 
agreement are not discussed. For any judgment for 
which reviewer inconsistency is found to exist, 
reviewers cite the evidence to support their 
conclusion. The evidence must relate to the criteria 
used to make the judgment and may not be the 
reviewer’s opinion or impression. In this fashion, the 
review process draws upon the deep expertise of the 
reviewers but constrains judgment by causing the 
reviewer to reference both evidence and decision 
criteria when a judgment is found to be inconsistent 
with other reviewers. 

The CCCA study used two content reviews to address 
the first research question: What are the prerequisite 
KSAs in mathematics and reading to qualify for entry- 
level, credit-bearing courses that satisfy general 
education requirements? Table 8 summarizes the 


processes of these content reviews using convergent 
consensus. 

The end result of the convergent consensus process 
was the creation of content maps. Content maps were 
created for mathematics overall and for each course 
title at the content area, subtopic, and objective level. 
Content maps were created for reading overall and 
for each course title at the content domain, standard, 
and objective level. The content maps are 
spreadsheets that summarize the groups’ consensus 
ratings on the prerequisite NAEP-specific KSAs and 
prerequisite KSA exclusions. The spreadsheets 
included four percentages for each NAEP-specific 


KSA that captured the percent of course packets in 
which the NAEP-specific KSA was (1) not 
prerequisite, (2) prerequisite, (3) prerequisite and 
important, or (4) consensus was not reached during 
group review. The content maps and lists of non- 
NAEP additional KSAs were provided to the NAEP 
experts for the final phase of the study. 

The study employed additional measures in order to 
examine reliability and validity of the convergent 
consensus process. These included process 
evaluations and a generaliz ability study. These are 
described later in this section. 


Table 8. Overview of Content Review Convergent Consensus Process 



Independent review 

Group review 

Participants 

• Reading content reviewers (12) 

• Alternate reading content reviewers (3) 

• Mathematics content reviewers (12) 

• Alternate Mathematics content reviewers (3) 

• Reading content reviewers ( 1 2) 

• Mathematics content reviewers (12) 

• NAEP reading experts (2) 

• NAEP mathematics experts (2) 

• EPIC facilitators (5) 

• EPIC scribes (5) 

• EPIC Principal Research Scientist and 

Assistant Research Scientist (2) 

• EPIC Project Director (1) 

Process 

1 . Holistic review training and holistic review of 
operational and validation course packets 

2. NAEP-framework training and review of 
training course packet(s) 

3. Qualifying process 

4. Web-based submission of ratings of NAEP 
objectives and non-NAEP additional KSAs 

1 . Onsite training 

2. Reading and mathematics reviewers work 
separately 

3. Four subject area review teams work 
separately 

4. Web-based submission of ratings 

5. Individual procedural feedback 

Review activities 

1 . Review course packet to identify and record 
evidence of NAEP-specific KSAs and non- 
NAEP additional KSAs 

2. Code the NAEP-specific KSAs and identify 
any relevant KSA exclusions using a web- 
based tool 

3. Code any non-NAEP additional KSAs 
identified in the course packet but not 
included in the NAEP-specific KSAs using a 
web-based tool 

1. Discuss and adjudicate all discrepant coding 
of NAEP-specific KSAs, including KSA 
exclusions 

2. Discuss all identified non-NAEP additional 
KSAs and code for each course packet 

Content review 

outcome 

Comprehensive list of prerequisite KSAs to address CCCA Research Question 1 (NAEP-specific 
KSAs and non-NAEP additional KSAs) 




Review Process 

The content review phase of the CCCA study 
consisted of two distinct reviews: independent review 
and group review. 

NAEP expert advisory panel. Prior to content 
review, the mathematics and reading NAEP experts 
convened to offer insights into and recommendations 
for the planned process, data collection, and 
outcomes. (See Appendix G for a meeting agenda.) A 
primary goal was for the research team to obtain 
guidance on study methods and to establish a NAEP 
framework-based coding scheme, which they applied 
to the 12 packets that would be coded by all reviewers 
within a subject. These included two training, two 
qualifying, and eight validation packets each in 
mathematics and reading. The NAEP experts were 
uniquely qualified to serve in this advisory role 
because they each have extensive expertise, judgment, 
and knowledge regarding college-level mathematics 
and reading demands in addition to depth of 
knowledge regarding the NAEP frameworks. The 
research team used the packets coded by the NAEP 
experts to assess the content reviewers’ ability to code 
the packets using the NAEP framework during the 
training and qualifying periods. 

The NAEP experts’ recommendations were used to 
ensure that course artifact reviews were conducted 
through an evidence-based process with procedural 
and internal validity. The experts’ review of course 
packets provided guidance to the research team to 
finalize the artifact collection decision rules related to 
textbooks, the coding scheme, and training for the 
content review. The experts’ codes of the training, 
qualifying, and validation packets were used in both 
the training and content review. 


Review process recommendations. NAEP experts 
reviewed the training materials, decision mles, and 
other key documents prior to the research team 
training the content reviewers and facilitators. Based 
on their review of 12 course packets, the advisory 
panel for each subject area, reading and mathematics 
respectively, provided recommendations relating to 
the following: 

• Finalizing the coding scheme 

• Finalizing the decision rules to support 
correct interpretation and application of 
NAEP objectives 

• Suggesting which course packets became the 
training, qualifying, and validation packets 

• Suggesting coding and threshold to evaluate 
content reviewer readiness to participate in 
operational packet reviews 

• Suggesting appropriate use of implicit 
evidence to identify prerequisite KSAs 

• Suggesting improvements to the global, or 
general, decision rules and subject area- 
specific decision rules 

• Suggesting improvements to the training 
processes and materials for the content 
reviewers and group review facilitators 

• Providing insight into the data sufficiency of a 
course packet, including assistance with 
identifying course packet textbook artifacts 

• Providing insight into the best approach to 
the content reviewer process of course 
packets, including use of course packet 
artifacts and artifact components (such as 
learning objectives on a syllabus or review 
section in a textbook) in identifying evidence 
of prerequisite KSAs 

• Providing insight into estimated time to 
complete all review tasks 


Subsequent to the content reviews, the NAEP experts 
convened again to review study findings and support 
the researchers’ interpretations of findings. 

A list of the implemented changes from the NAEP 
advisory panel is contained in Appendix H. 

Content Review Group Composition and Course 
Packet Assignment 

A total of 32 content reviewers participated in the 
CCCA study. In preparation for the study, the 
content reviewers were divided into four mathematics 
and four reading content review groups, each 
composed of four content reviewers — three reviewers 
plus one alternate. The alternate content reviewers 
underwent the same training and review process as 
their teams during the holistic and independent 
review. Alternates participated in the group review 
process when a reviewer was unavailable to attend the 


Content Reviewer Demographics 

Education 

58% doctorate 

42% master’s 

Occupation 

Years 

Teaching in 
Higher 

75% university faculty 

1 3% community college faculty 

8% education consultant 

4% other education-related occupation 

Education 

7 1 % more than 1 0 years 

Gender 

54% male 

46% female 

Race 

83% White 

4% Black/African American 

4% Asian or Pacific Islander 

9% No response 


group review meeting. Content reviewers were 
assigned to content review groups to ensure an even 
distribution of two- and four-year instmctors and 
experienced reviewers. 

For each course title in mathematics or courses with 
substantial reading demands, there were 22 course 
packets associated with each of the four course titles. 
Two of the course packets in each course title were 
validation packets that were coded by all content 
reviewers in the course subject area and which 
provided data for the reliability and generalizability 
analyses. The other 20 course packets in each course 
title were considered operational packets and were 
distributed randomly for coding among the four 
groups of content reviewers in the appropriate subject 
area. Each content reviewer coded five operational 
packets and two validation packets from each course 
title within a subject area, for a total of 28 course 
packets per group. Reviewers were not informed as to 
whether their packets were classified as validation or 
operational. Figure 3 summarizes the details of the 
operational and validation course packet assignments 
for mathematics; an identical process was used for the 
assignment of course packets for courses with 
substantial reading demands. 

Course packets were used for different purposes, by 
different study participants, at different points in time. 
To summarize, there were 184 packets in total. The 
NAEP experts used the NAEP frameworks to 
precode 24 packets: four training, four qualifying, and 
sixteen validation packets. The eight training and 
qualifying packets were used for training and were not 
used for any other purpose. The sixteen validation 
packets were coded, using the NAEP frameworks, 
across the content reviewer groups so they could be 
used as a basis for the generalizability study to 
establish intra- and interrater reliability. Finally, the 
operational packets were coded and provided the data 
for developing the content maps and establishing the 



Figure 3. Operational and Validation Course Packet Assignments 


prerequisites KSAs. Table 9 provides a summary of 
the review process. 

Overview of Content Reviewer Training 
EPIC developed a series of trainings to introduce 
content reviewers to the CCCA study, which 
incorporated the guidance of the NAEP advisory 
panels. Training webinars oriented the content 
reviewers to the main goals of the study and the 
concept of a prerequisite KSA, then guided the 
reviewers to refer to evidence in the course packets to 
complete content reviews. (See Appendix I for 
holistic review orientation and independent review 
training materials.) Training packets were used to 
determine areas for focused individualized training by 
the research team as deemed necessary or requested 
by content reviewers. 

Content reviewers could comment or provide 
feedback on the decision rules twice during the study. 
The first time was during training, and the second was 
during independent review. During training, feedback 
identified needs for clarifications that were provided 
to everyone and documented for independent and 
group review. During independent review, feedback 


was monitored for clarifications that might be 
necessary to ensure consistent application of the 
coding scheme. No changes were made to the 
decision mles after training. 

Definition of prerequisite knowledge, skills, and 
abilities. The content reviewers’ main task was to 
identify the prerequisite KSAs based on evidence 
found in the course packets. This study focused on 
the prerequisite KSAs a student needs to be prepared 
to enter a credit-bearing college course. Content 
reviewers were trained to consider whether there was 
evidence of a KSA within the course packets, and 
then to determine whether the KSA was prerequisite. 
The following CCCA definition of prerequisite was 
provided to content reviewers. 


A KSA is prerequisite if a minimally prepared 
student is either expected or required to possess 
this knowledge, skill, or ability to be prepared for 
entry into the course. A prerequisite KSA may be 
reviewed but not taught in depth or for the first 
time during the course. 


Holistic review. A holistic review of course packets 
was conducted before the independent review and 
prior to introducing content reviewers to the NAEP 
framework. The purposes of the holistic review were 
to develop familiarity with: the course packets, 
application of global decision rules, identification of 


sources of evidence, annotation techniques, and use of 
the web-based survey instrument. A product of this 
holistic review is a list of holistic review statements. 
Although they are the product of a less developed 
understanding of the content review, the holistic 
review statements are free of any bias that might be 


Table 9. Summary of Review Process 






Independent 



Advisory panel 

Holistic review 

Training session 

review 

Group review 


NAEP experts 
precoded the 

All content 

All content 

All content 

Content reviewers 


training, qualifying, 

reviewers, 

reviewers, 

reviewers passing 

selected for group 

Number of 

and validation 

including 

including 

qualification, 

review 

packets 

packets 

alternates 

alternates 

including alternates 

participation 

Training: 

2 mathematics, 

2 reading 

Precoded using the 
NAEP frameworks 
by NAEP experts 
to establish 
training materials 

NA 

Coded using NAEP 
frameworks during 
training and 
coaching sessions 

NA 

NA 


Precoded using the 


Coded using NAEP 



Qualifying: 

NAEP frameworks 


frameworks after 



2 mathematics, 

by NAEP experts 

NA 

training to qualify 

NA 

NA 

2 reading 

to establish 


for Independent 




qualifying materials 


Review 



Validation: 

8 mathematics, 

8 reading 

Precoded using the 
NAEP frameworks 
by NAEP experts 
to establish 
reference materials 

Reviewed for 

KSAs without 
reference to NAEP 
frameworks 

NA 

Coded 

independently 
using NAEP 
frameworks 

Reviewed coding 
of all packets, 
including 
exclusions and 
Non-NAEP KSAs 

Operational: 

80 mathematics, 

80 reading 

NA 

Reviewed for 

KSAs without 
reference to NAEP 
frameworks 

NA 

Coded 

independently 
using NAEP 
frameworks 

Reviewed coding 
of all packets, 
including 
exclusions and 
Non-NAEP KSAs 

Summary by subject area 

Mathematics or 
reading packets 
reviewed per 
group 

Each team of 

Individual 

Individual 
reviewers coded 

Individual 

Individual 

NAEP experts 
coded all of the 
same 1 2 packets in 
their subject area 

reviewers 

reviewed 28 
packets holistically: 

8 validation, 

20 operational 

using the NAEP 
frameworks up to 
a maximum of 4 
packets: 2 training, 

2 qualifying 

reviewers coded 

28 packets in four 
teams: 8 validation, 
20 operational 

reviewers 

reviewed the 28 
coded packets: 

8 validation, 

20 operational 

Total packets 

Total of 1 84: 

T otal of 24: 

Total of 1 84: 

T otal of 8: 

Total of 1 76: 

Total of 1 76: 

92 mathematics, 

1 2 mathematics, 

92 mathematics. 

4 mathematics, 

88 mathematics. 

88 mathematics. 

92 reading 

1 2 reading 

92 reading 

4 reading 

88 reading 

88 reading 






introduced by knowing the NAEP framework and they 
serve as a resource for generation of non-NAEP 
additional KSAs during the content review. 

The holistic review orientation was conducted to 
introduce the content reviewers to the CCCA study, 
course packets, and the concepts of evidence and 
prerequisite KSAs. The request to each content 
reviewer was to review the 28 course packets, in a 
summary fashion, and to identify four to ten potential 
prerequisite KSAs per packet. There were very few 
restrictions on the definition of the prerequisite 
KSAs, and the resulting holistic review statements 
were at vasdy different levels of specificity. 
Duplication of holistic review statements, from 
packet to packet and across reviewers, was expected. 
Reviewers noted sources of evidence for each holistic 
review statement so that the information would be 
available for the subsequent independent and group 
review cycles. 

An online form was used to facilitate the collection of 
holistic review statements identified during the holistic 
review. The holistic review instrument collected course 
packet identifying information and four to ten holistic 
review statements in open-ended textboxes. The 
following instructions were provided to content 
reviewers for the holistic review of course packets. 


Review each packet in its entirety to obtain a holistic 
understanding of the course. Note all prerequisite 
knowledge, skills, and abilities, and annotate them 
clearly as you go. Remember that you will need to 
refer to your notes here when you meet with the 
group later on. The more clear and detailed your 
notes are, the more helpful they will be to you. In 
your own words, describe 4-10 prerequisites to this 
course below. 


Prior to implementation of the online form, the 
research team completed multiple test cases to verify 
its ease of use and the ability to ensure data integrity. A 
link to the holistic review form was released to content 
reviewers after testing. Content reviewers completed 
the holistic review of the course packets in two weeks. 
See Appendix J for the holistic review instrument. 

All annotated evidence of holistic review statements 
were considered for inclusion during independent 
review as either a non-NAEP additional KSA or a 
NAEP-specific KSA. Content reviewers determined 
whether any of the holistic review statements they 
generated should be proposed as either NAEP- 
specific KSA or non-NAEP KSA. Differences 
between the processes of the holistic review and the 
independent review resulted in holistic review 
statements not being carried forward into the 
independent review. Many of the holistic review 
statements were not of similar specificity as those 
identified during independent review, when the 
reviewers had the NAEP framework as a reference. 
Also, evidence and the process for identifying 
evidence became more highly defined as the reviewers 
were transitioned from the holistic review to the 
independent review phase of the study. Therefore, 
holistic review statements, while they may serve as 
resources for the independent review, were not 
intended to be directly comparable to the content 
review KSAs. 

Figure 4 displays the total number of holistic review 
statements generated during the holistic review. These 
numbers include all duplications, those with very 
similar content, those later determined to map to the 
NAEP frameworks (NAEP-specific KSAs), those 
later identified as non-NAEP additional KSAs, and 
those beyond the scope of this research study, i.e., not 
relevant to mathematics or reading. 
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During the independent review training, reviewers 
were encouraged to consider their holistic review 
evidence notations and the holistic review statements 
as they conducted the independent review of the 
same 28 course packets. Some of the holistic review 
statements identified by the reviewers mapped to a 
NAEP framework statement or were deemed by the 
reviewer to be a non-NAEP additional KSA 
candidate. Other holistic review statements, and 
related evidence, were not carried forward because 
they were deemed by the reviewer to be outside the 
scope of the study as a result of introduction of new 
parameters during the independent review (decision 
rules, definitions of evidence, etc.). 
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Figure 4. Holistic Review Statements Generated 
During Holistic Review 


Training reviews. Content reviewers received web- 
based training as a content review group and then 
individually practiced coding course packets using the 
independent review instmment. The training packets 
were representative of those that content reviewers 
would encounter during the independent review. Prior 
to coding training packets, content reviewers attended 
one of four orientation webinars where they were 
instructed in the purposes of the study, the process and 
decision mles for reviewing course packets, and the 
coding scheme for the course packets. Three content 
reviewers did not attend a training webinar but 
reviewed the archived webinar materials. 


Note. This figure represents non-NAEP additional KSAs 
containing duplicative or redundant content. 

Independent review. The second training, the 
independent review training, introduced the content 
reviewers to the NAEP framework. The independent 
review training also introduced a coding scheme, 
decision mles, definitions of terms specific to the 
NAEP framework, and an expanded definition of 
what constituted evidence when discerning whether a 
statement was evidence of a KSA. See Appendix K 
for the independent review submission instrument. 


After orientation, content reviewers received a 
training packet that they coded independently. They 
then assessed how their codes aligned to precoded 
responses provided by the NAEP experts during the 
NAEP advisory panel meetings. Content reviewers 
had an opportunity to complete up to two training 
packets and discuss any discrepancies in their coding 
with each other and the research team. 

Qualifying reviews. After a content reviewer completed 
a training packet(s), he or she coded a qualifying packet. 
The research team compared content reviewers’ coded 


qualifying packets with precoded responses provided by 
the NAEP experts. Table 10 displays the five conditions 
that resulted in a KSA objective match. There was no 
distinction made between a KSA coded as 
“prerequisite” and one coded as “prerequisite and 
important.” Consistently locating prerequisite KSAs 
using identifiable evidence from course packets was 
more important for ensuring content reviewers mapped 
to die process than the distinction between 
“prerequisite” and “prerequisite and important.” 

Content reviewers qualified for participation in the 
study if they aligned 80% of their codes with the 
precoded responses provided by the NAEP experts 
during the NAEP advisory panel. If content reviewers 
did not meet the 80% threshold they had the 
opportunity to repeat training and attempt a second 
qualifying packet. Content reviewers were removed 
from the study if they did not reach the 80% 
threshold on their second qualifying packet based on 
the coding alignment noted in Table 10. 

Alternate reviewers for the content review groups were 
identified by having lower levels of (but still acceptable) 
alignment on their qualifying packets as compared to the 


Table 10. Qualifying Review Conditions Resulting in KSA 
Objective Match 


Condition 

Content 
reviewer code 

Precoded 

response 

1 

KSA is NOT 
PREREQUISITE 

KSA is NOT 
PREREQUISITE 

2 

KSA is 

PREREQUISITE 

KSA is 

PREREQUISITE 

3 

KSA is 

PREREQUISITE 

KSA is 

PREREQUISITE and 
IMPORTANT 

4 

KSA is 

PREREQUISITE and 
IMPORTANT 

KSA is 

PREREQUISITE 

5 

KSA is 

PREREQUISITE and 
IMPORTANT 

KSA is 

PREREQUISITE and 
IMPORTANT 


other content reviewers in their group. See Appendix L 
for training and qualifying results. 

Independent Review 

Each content reviewer independently coded 28 course 
packets. The content reviewers used the independent 
review data collection instmment to record prerequisite 
NAEP-specific KSAs, KSA exclusions, and non- 
NAEP additional KSAs for each course packet. 
Content reviewers began their review by reading all 
material provided in the course packet and noting 
where they found evidence of prerequisite KSAs. 

Content reviewers coded both the applicability and 
importance (for those KSAs that were found to be 
prerequisite to the course) of each NAEP-specific 
KSA. (The instructions provided to content reviewers 
are shown in the first box on the next page.) 

During independent review, content reviewers 
manually entered prerequisite KSA exclusions when 
some but not all components of a NAEP-specific 
KSA were considered prerequisite to the course. 
Within the independent review instrument, a code 
was required for all NAEP-specific KSAs; NAEP- 
specific KSA exclusions were optional. (The 
instructions provided to content reviewers for 
excluding parts of non-NAEP additional KSAs are 
shown in the second box on the next page.) 

For the purpose of analyzing the course packet 
sufficiency, the content reviewers were asked to 
provide a code for each course packet on the 
helpfulness of each type of artifact, including the 
syllabus, assignment(s), assessment(s), text excerpt, 
study guide/review, instructor comments, homework 
assignments/readings, and other artifacts (not included 
in the list). The following options were provided to 
content reviewers for coding the relative helpfulness of 
the different course artifacts: not applicable, not 
helpful, limited helpfulness, helpful, and very helpful. 




For all NAEP framework objective KSAs (NAEP-specific KSAs), please indicate whether each is a prerequisite for this 
course or not, and if so, how important it is to preparedness for this course. A KSA is prerequisite if a minimally 
prepared student is either expected or required to possess this knowledge, skill, or ability to be prepared for 
entry into the course. A prerequisite KSA may be reviewed, but not taught in depth or for the first time, during the 
course. 

1. KSA is NOT A PREREQUISITE for this course. There is no evidence that this is a prerequisite (e.g., there is no 
evidence of the KSAs described by this objective in the packet or this is a new skill or ability that will be taught 
in this course.) 

2. KSA is PREREQUISITE for this course. Without this prerequisite, students may struggle in some areas of this 
course. 

3. KSA is PREREQUISITE for this course and is IMPORTANT. Without this prerequisite, students will not be 
prepared for this course and will struggle in this course. 


Many times, part(s) of a framework objective KSA is/are prerequisite to a course, while other parts are not 
prerequisite. For each framework objective KSA where only part of the objective is prerequisite, please list the parts 
that are not. 

For example, an exclusion might look like this when organizing structures only need be located, but not recalled, 
and these structures include comparison/contrast and problem/solution, but not enumeration: 

Locate or recall organizing structures of texts, such as comparison/contrast, problem/solution , enumeration, etc. 

And would be recorded like this: 

Recall, enumeration 

If any part of an entire objective statement is prerequisite, while other parts are not, list the parts that do not apply 
in the text box. 


Finally, for each course packet, the content reviewers 
entered prerequisite non-NAEP additional KSAs, if 
present, on the independent review instrument. 
Content reviewers were asked to identify non-NAEP 
additional KSAs if there was any evidence that 
appeared to be prerequisite to a course packet but not 
described by the NAEP frameworks. 

For each prerequisite non-NAEP additional KSA 
provided, content reviewers annotated the evidence 
and provided the source of evidence (syllabus. 


assignment, assessment, text excerpt, etc.) they used 
to generate the prerequisite non-NAEP additional 
KSA on the independent review instrument. (The 
instructions provided to content reviewers for 
entering non-NAEP additional KSAs are shown on 
the next page.) 

As noted earlier, content reviewers were expected to 
use the evidence annotated during the holistic review 
of the course packets as a starting point for their 
independent review of the same course packets and 


determine whether any of the holistic review 
statements they generated should be proposed as 
either a NAEP KSA or a non-NAEP additional KSA. 


Since content reviewers in the same groups 
conducted reviews against the same set of packets, 
redundancy within the non-NAEP additional KSAs 
occurred. Since this was an interim step of the 
convergent consensus process, redundancies within a 
group were consolidated during group review. 
Redundancies across groups were consolidated as the 
non-NAEP additional KSAs were compiled for the 
NAEP expert review and further at the NAEP expert 
review. Figure 5 summarizes the number of non- 
NAEP additional KSAs generated during the 
independent review. 

The following circumstances explain why some KSA 
statements identified during the holistic review were 
likely not included in the non-NAEP additional KSA 
list generated during independent review. 

• Independent review training contributed to 
content reviewers’ deeper understanding of 
prerequisite. For example, some holistic 
review statements were found to be taught or 
reviewed content in the course and, therefore, 
not prerequisite. 

• Independent review training contributed to 
content reviewers’ deeper understanding of 
evidence. Decision rules were developed for 
independent review, including how to identify 
evidence. Some annotated evidence of KSAs 
in the holistic review, therefore, did not meet 
the level of accepted evidence for the 
independent review. 

• Independent review training helped develop a 
common understanding of terminology. 
Content reviewers held different opinions on 
how to define broad terms prior to 
independent review training. During the 


Please list any additional prerequisites that you 
did not identify in your initial listing of 
prerequisites and were not contained in the 
NAEP objectives. 

For each additional prerequisite knowledge, skill or 
ability evident in this course packet, please identify in 
which artifact the evidence was found. Please select 
all that apply. 
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Figure 5. Non-NAEP Additional KSAs Generated 
During Independent Review 

Note. This figure represents non-NAEP additional KSAs 
containing duplicative or redundant content. 


holistic review, the term “basic algebra” might 
have been interpreted to mean knowing how 
to perform arithmetic operations by one 
reviewer, while another reviewer might have 
interpreted “basic algebra” to include solving 
polynomial equations. The NAEP 
frameworks defined many terms used in the 
independent review. 

• Independent review training contributed to 
content reviewers’ common understanding of 
the grain size of a KSA. Content reviewers did 
not consider a holistic review statement such 
as “basic algebra” to be a sufficient match for 
a NAEP objective statement such as “Create 
and translate between different 
representations of algebraic expressions, 
equations, and inequalities (e.g., linear, 
quadratic, exponential, or *trigonometric) 
using symbols, graphs, tables, diagrams, or 
written descriptions (NAEP objective 5.2.a).” 

Appendix M contains the reference sheet for 
reviewers to complete coding of the content review. 

Reviewer agreement. The research team constructed 
aggregated score reports to represent the number and 
type of KSA coding discrepancies within each content 
review group prior to the group review. Each NAEP- 
specific KSA had potentially two types of KSA 
coding discrepancies: (1) KSA applicability discrepancies 
(not prerequisite, prerequisite, or prerequisite and 
important), and (2) KSA exclusion discrepancies (when 
applicable). The aggregated score report served as the 
framework for discussion during the group review. 
Any packet with less than 100% agreement on KSA 
applicability and prerequisite KSA exclusions required 
group discussion. There were 2,072 possible points of 
discrepancy for the reading groups' 1 and 7,280 for the 


6 28 packets per reading group x 37 reading framework statements x 2 types of 
possible KSA coding discrepancies. 


mathematics groups. ' The prerequisite non-NAEP 
additional KSAs identified by content reviewers were 
carried forward and discussed at the group review. 

Group Review 

After the independent review, teams of content 
reviewers participated in the group review meetings. 
At the beginning of the group review meetings, the 
research team oriented the reviewers to the process, 
decision mles, and expected outcomes of the study. 
Each content review group’s task during the 4-day 
meeting was to come to consensus on the KSA 
applicability discrepancies and KSA exclusion discrepancies as 
identified by the aggregated score reports. Content 
review groups had an opportunity to revisit KSAs 
where consensus was reached during independent 
review when the interpretation of decision rules, 
terminology, or evidence required a coding change. 

The research team eliminated duplicates prior to 
group review and each content reviewer group had an 
opportunity to combine, eliminate, and revise the list 
of non-NAEP additional KSAs prior to discussing 
coding discrepancies. The content review groups 
discussed each prerequisite non-NAEP additional 
KSA to reach consensus on an applicability code after 
reviewing the NAEP-specific KSA discussion points 
for each course packet. For some groups, the lists 
were pared down as they moved through their 28 
packets. If there was a reason to revisit a previous 
packet, the same process was used as was developed 
for NAEP-specific KSAs. The outcome of the group 
review included a list of non-NAEP additional KSAs 
that were identified as prerequisite to the course 
packets reviewed by each group. The full list of 
prerequisite non-NAEP additional KSAs discussed at 
group review for both mathematics and reading is 
displayed in Appendix N. See Appendix O for the 
group meeting agendas. 


7 28 packets per mathematics group x 130 mathematics framework statements x 
2 types of possible KSA coding discrepancies. 


Facilitators, scribes, and oversight. EPIC facilitators, 
experienced in group facilitation and the convergent 
consensus process and trained in CCCA methodology, 
led each review team in the discussion of KSA coding 
discrepancies during the group review meetings. 

An EPIC-trained scribe assisted each facilitator and 
was responsible for recording all the decisions made 
by the content review groups using a group review 
instrument. All facilitators and scribes were trained 
together to ensure that they were highly 
knowledgeable about the purposes, goals, and design 
of the study, and also were instmcted in protocols to 
ensure that decision rules were applied consistently 
across all content review groups. This training 
included mock reviews of course packets so that 
facilitators had direct experience with the packets, 
applying the same processes and decision rules that 
content reviewers used during the group review. 
During the training, scribes were acquainted with the 
group review instrument. 8 See Appendix P for the 
facilitator and scribe training overview. 

Group review coding scheme. During the group 
review, scribes recorded content review group answers 
to a series of questions related to the results of the 
independent review. Scribes also recorded the final 
consensus ratings on coding discrepancies. The end 
result was a final content review group code for each 
NAEP-specific KSA. In their last task, each group 
attempted to reach consensus on the applicability of all 
prerequisite non-NAEP prerequisite KSAs recorded 
during independent review for each of the 28 course 
packets. The coding scheme for each question type is 
described in Table 1 1 . 

In addition to scribes and facilitators, the principal 
research scientist (PRS), the assistant research 
scientist (ARS), and the project director (PD) 


attended the group review meetings. The role of the 
PRS and the ARS was to closely monitor the group 
interactions, resolve issues, and ensure that the groups 
followed the procedures for the review meetings. The 
PD managed any administrative issues that arose. 

Two NAEP experts also attended the meetings to 
answer questions regarding the NAEP frameworks 
and to provide oversight. The participation of the 
NAEP experts during this phase provided additional 
insight and input for the next set of NAEP meetings 
where the content maps were reviewed and analyzed. 

Phase 3: Conducting the NAEP Expert 
Review 

A panel of mathematics and a panel of reading NAEP 
experts reviewed the outcomes of the group content 
reviews, evaluated the prerequisite KSAs, conducted a 
series of comparisons of the prerequisite KSAs to the 
NAEP framework and other preparedness research, 
and provided input to the research team on the 
interpretation of findings. See Appendix Q for the 
NAEP expert review meeting agendas. 

The instructions below were provided to the NAEP 
experts at the beginning of the meetings concerning 
process flexibility. 

Table 12 outlines the processes of NAEP expert 
multiple reviews. 

The NAEP experts consulted the content maps of 
KSAs to generate narrative descriptions that 
summarized the KSAs necessary for students to be 


The exact process to use will be determined by 
what works best for your team. For example, you 
may complete all tasks for one course type and then 
move through each course type individually, or you 
may complete the first task for college overall and 
then complete the second task for college overall. 


Rounds of end-to-end testing that accounted for all possible question triggers 
and branching scenarios were completed before the group review meetings. 


Table I I. Group Review Coding Scheme 


Question Coding scheme 


Was consensus reached for this KSA during 
independent review? 

1. Yes 

2. No 

3. Yes, but group revisited this KSA and changed coding 

Was consensus reached for this KSA during 
group review? 

1. Yes 

2. No 

What was the group’s final rating for this KSA? 

1 . KSA is not prerequisite 

2. KSA is prerequisite 

3. KSA is prerequisite and important 

4. Final rating not established during group review 

List the portions of the KSA excluded by the 
group, if any. 

Open-ended text 

Who in the group changed his/her mind? 

1. Member One 

2. Member Two 

3. Member Three 

4. None 

Why did Member One/Two/Three change 
his/her mind? 

1 . New shared understanding of the objective 

2. Other group member(s) made convincing argument 

3. Other group member(s) disputed evidence 

4. Evidence not seen before or changed understanding of evidence 

5. Changed understanding of terminology 

6. Technical error with survey tool 

7. Changed understanding of application of decision rule 

8. Other 

Does this new understanding of the application 
require recoding the packets? 

1. Yes 

2. No 

Why was consensus not reached in the group 
review? 

1 . Member One was not persuaded by other two 

2. Member Two was not persuaded by other two 

3. Member Three was not persuaded by other two 

4. All three members had different coding and were not persuaded by 
one another 

5. Other 


academically prepared, without remediation, for entry- Step One: Create a narrative description of the ( mathematics or 
year college-level mathematics courses and courses reading) knowledge, skills, and abilities that are prerequisite for 

with substantial reading demands. The content maps college courses, based on the evidence from the college course 

are summarized in the Results section, and the content analysis. The experts first reviewed the content 

objective-level content maps are shown in Appendix R. maps containing the prerequisite and prerequisite and 

important NAEP framework objectives for each 


The NAEP experts then proceeded using the 
following steps: 

course and for mathematics and reading overall. This 
review also included the KSA exclusions and the non- 




Table 12. NAEP Expert Review Process 


NAEP review process 

Participants 

• NAEP reading experts (3) 

• NAEP mathematics experts (3) 

• EPIC Principal Research Scientist and Assistant Research Scientist (2) 

• EPIC Project Director (1) 

• Facilitation consultant (1) 

Process 

• In-person, onsite, 3-day meeting 

• Training onsite 

• Procedural feedback 

Review activities 

• Review list of KSAs and content map for course type containing each course title 

• Develop narrative descriptions based on the content map for each course type 

• Compare the narrative descriptions of the prerequisite KSAs to the following: 

o Description of minimal requirements for placement into college-level coursework from the JSS 
research and determine the commonalities and differences in skill requirements 
o Grade 1 2 NAEP 2009 and 20 1 3 item pools to determine the content of NAEP that is or is not 
consistent with the prerequisite KSAs 


o KSAs represented by 2009 items that map to the NAEP score scale with a response probability of 
.67 and fall within the range of cut scores set by the two replicate panels in the JSS research for 
college to determine the commonalities and differences in content requirements 
o KSAs represented by 2009 items that map in the range of the NAEP score scale from the beginning 
of the Basic level to an emerging stopping point 

o NAEP achievement level descriptions 

Review outcome Summary document to address CCCA Research Questions 2-4 


NAEP additional KSAs. The experts reviewed the 
content maps and then, as a panel, synthesized and 
interpreted them based on their expertise with the 
NAEP frameworks and their in-depth understanding 
of mathematics and reading content. Using the 
prerequisites extracted from the artifacts — including 
the KSA exclusions and non-NAEP additional 
KSAs — they drafted a narrative description of the 
mathematics and reading content that is necessary for 
students to be academically prepared, without 
remediation, for entry-year college-level mathematics 
and reading. 

Given the differences in the structure of the NAEP 
frameworks between reading and mathematics, and 


the difference in independence 5 of course titles 
between the subject areas, the two NAEP expert 
groups had flexibility in how they determined whether 
a prerequisite KSA was incorporated into the 
narrative description. The NAEP reading experts 
selected a threshold of two thirds: a prerequisite KSA 
was required by the content review groups to be 
present in 67% of course packets for inclusion in the 
narrative description. The NAEP mathematics 
experts used an 80% criterion for determining 
whether to include the prerequisite KSA in the 
narrative description. 


9 The titles reviewed by the reading experts were independent of one another — 
any or all courses could be taken in the same school term. Some of the 
mathematics courses were sequential, e.g., college algebra would be taken prior to 
precalculus/calculus, although both are entry-level courses depending on the 
prior mathematics performance of the student. 




Step Two: Compare the prerequisite knowledge, skills, and 
abilities in the narrative description to the knowledge, skills, 
and abilities measured by the NAEP items. After 
synthesizing the prerequisite NAEP-specific KSAs, 
KSA exclusions, and the non-NAEP additional KSAs 
into the narrative description, the NAEP experts 
reviewed the 2009 and 2013 item pools. They 
identified the range on the NAEP scale that 
corresponded to the prerequisites in mathematics and 
reading narratives. The NAEP experts reviewed each 
item to identify the KSAs assessed, and then 
determined if a student with the prerequisite KSAs 
described by the narrative would be likely to answer 
each item correctly. They started at the lowest 
difficulty item and worked through the items in order 
of difficulty. As soon as items assessed KSAs above 
those identified in the narrative, the experts noted the 
corresponding point or range on the NAEP scale. 
Items that assessed KSAs not identified as 
prerequisite to any course were skipped, and if this 
resulted in a large gap between items, the experts 
noted the corresponding scale range. As they engaged 
in this process, the NAEP experts reviewed the items’ 
content in relation to prerequisite KSAs identified in 
the narrative description and the evidence (e.g., 
textbook samples, assignments) used to make the 
prerequisite determination. 

The reading experts were confident in collapsing 
prerequisites across all courses reviewed by the reading 
content reviewers; the mathematics experts noted 
differences between courses that necessitated reporting 
findings by course, rather than for mathematics overall. 
This arose from the sequential nature inherent in the 
mathematics courses (e.g., college algebra would be 
taken prior to calculus) versus the lateral equality of the 
courses identified as having substantial reading 
demands (e.g., a first-year student could take all four 
course titles in the same semester). 


Step Three: Compare the narrative descriptions to the 
Borderline Performance Descriptions (BPDs) developed for the 
JSS studies for college. The NAEP experts compared 
their narrative descriptions against the narrative BPDs 
created in the JSS study to determine how the results 
of the CCCA study compare with previous NAEP 
preparedness research. 

Step Four: Discuss how these prerequisites inform future 
NAEP preparedness research. In order to address the 
fourth research question of the CCCA study, the 
NAEP experts’ final task was to engage in a guided 
discussion on how the mathematics and reading 
prerequisites that were identified could inform future 
NAEP preparedness research. EPIC synthesized 
discussion points from throughout the meeting and 
developed questions in order to guide the NAEP 
experts in a discussion of their findings, how they 
compared with earlier NAEP preparedness research, 
and possible future directions for the NAEP. 

The following questions for discussion were 
presented to the NAEP experts: 

• What do each of you see as the most 
important implication or application of the 
prerequisite KSAs and their corresponding 
point on the NAEP scale? What are your 
recommendations for next steps? 

• Based on the prerequisite KSAs, what can we 
suggest as priorities for NAEP? 

• Based on the prerequisite KSAs, what can we 
suggest as priorities for high schools? 

• JSS was done before the implementation of 
the Common Core State Standards (CCSS). 
Did the lack of the CCSS affect the 
development of the JSS BPD? 

• What suggestions do you have that would 
inform future research efforts on the NAEP? 


Process Evaluations 

Both the content reviewers and NAEP experts 
completed process evaluations to address the 
procedural validity of the CCCA study. Procedural 
validity refers to evidence that the procedures 
employed in the study’s implementation were 
performed as intended and were understood by all 
participants. Content reviewers responded to three 
process evaluations: one after (1) the holistic and 
independent review trainings, (2) the independent 
review, and (3) the group review. The process 
evaluation questions replicated, or were modifications 
of, evaluations used in previous research (W estEd & 
EPIC, 2013). The NAEP experts completed a process 
evaluation after the advisory panel. Appendix S 
contains the process evaluation questionnaires. 

The first content reviewer process evaluation survey 
provided feedback regarding the holistic review and 
independent review training. Content reviewers 
completed this process evaluation after completing 
the independent review training. The evaluation 
questions related to the clarity and helpfulness of the 
training webinar and materials. Content reviewers 
were asked about their confidence in the training 
procedures to prepare them for their role and their 
understanding of their role in the CCCA study. 

The second content reviewer process evaluation 
survey was completed after the content reviewers 
completed their independent reviews. This evaluation 
captured process-related feedback regarding the 
independent review, content reviewers’ overall 
impressions of the process, their satisfaction with 
their work, what worked well, what did not work well, 
and how the process might be improved. 

The third content reviewer process evaluation survey 
was completed after the group review consensus 
meetings. This process evaluation focused on 
gathering information about the convergent 


consensus process. An important measure of 
procedural validity of the convergent consensus 
process is the evaluation of the process by the content 
reviewers themselves, with high degrees of 
satisfaction with the training and the process boding 
well for the likelihood that, if the process were 
replicated, similar results would be found. The 
reviewers indicated that their opinions were heard, 
they were not pressured to agree, and the process 
overall was fair. 

The NAEP experts were asked to complete a process 
evaluation after the advisory panels. This process 
evaluation gathered information about the process of 
the NAEP advisory panels and the NAEP experts’ 
understanding of their role in the study. In addition to 
the process evaluation, NAEP experts were asked 
questions for discussion to which they provided 
narrative responses at the conclusion of the advisory 
panel meetings and expert review meetings about 
their confidence in the output and procedures of the 
meetings. See Appendix S for the questions for 
discussion. 

Generalizability Study 

The generalizability study was designed and conducted 
to investigate the reliability of the methods and 
operational coding processes used in this study. The 
validation course packets, coded during the 
independent and group review processes, generated the 
data used in the eight generalizability studies (two in 
both mathematics and reading for both independent 
review and group review). Each generalizability study 
provided two generalizability (G) study coefficients and 
two decision (D) study coefficients. 

The results of the G- and D-studies contribute to the 
understanding of the validity of findings in different 
ways. The G-study provides evidence in the form of 
variance component estimations that measure how 
consistent content reviewers were in coding 


prerequisite KSAs. After training, content reviewers 
ideally did not vary much, if at all, in their ability to 
identify prerequisite KSAs. The findings of this study 
are valid and generalizable to the extent that content 
reviewers were consistent in coding course packets, as 
evidenced by the G-study coefficients. 

The D-study uses the variance component estimations 
from the G-study to provide evidence that the 
operational processes used to code course packets 
produced reliable findings. Estimates from the D-study 
indicate whether the operational coding process of 
using 20 course packets and three content reviewers 
during independent review and using 20 course packets 
at the group-level review produced acceptably low 
levels of variance in identifying prerequisite KSAs. If 
20 course packets produced too much variance, the D- 
study coefficients would be lower than .70 and more 
than 20 course packets would be required to produce 
reliable results. The same would be true of content 
reviewers and content reviewer groups. 

The independent review generalizability study entailed 
an analysis of a fully crossed two-factor content 
reviewer (12) x course packet (8) random effects design 
wherein all content reviewers in both mathematics and 
reading coded the same eight validation course packets. 
The dependent outcome variable indicated whether the 
content reviewer coded a NAEP-specific KSA as not a 
prerequisite, prerequisite, or prerequisite and important. 
Absolute ((|)) and relative (p) generalizability 
coefficients were estimated for the independent 
reviewer coding of the validation course packets. In 
addition to the analysis at the NAEP objective level of 
the KSAs, a separate set of generalizability coefficients 
were estimated at the NAEP subtopic or standard level 
of the KSAs by aggregating the objectives within each 
NAEP subtopic or standard. The NAEP subtopic or 
standard code was defined by the NAEP objective 
within the subtopic or standard that had the highest 
code {prerequisite and important was highest code. 


followed by prerequisite). 

The group review generalizability study was identical 
to the independent review generalizability analyses 
except that individual content reviewers were replaced 
by group consensus codes for each of the eight 
validation course packets. These group consensus 
codes were subsequently analyzed in a two-factor 
content reviewer group (4) x course packet (8) fully 
crossed random effects design to estimate absolute 
and relative generalizability coefficients at both the 
NAEP objective and subtopic or standard level. 

Moreover, all generalizability analyses included a set of 
decision study (D-study) estimates designed to estimate 
the reliability of the operational coding process. Three 
content reviewers during the independent review and 
one group at the group review coded 20 operational 
course packets. These values were used in the D-study 
coefficient estimation procedure. 

In summary, a total of eight generalizability analyses 
were conducted for this study. Each analysis provides 
G-study absolute ((j)) and relative (p) coefficients as 
well as operational D-study absolute (())) and reladve 
(p) coefficients. The absolute g-coefficient ((j)) is 
considered an “index of dependability” and is useful 
when one is concerned with the exact value of the 
score, regardless of rank ordering (Brennan, 2001; 
Webb & Shavelson, 2005). The relative g-coefficient 
(p) is analogous to the reliability coefficient in 
classical test theory and is useful when concerned 
with the rank ordering of elements or respondents. 
Thus, for the full study, 32 generalizability coefficients 
are provided. In general, a coefficient of .70 is an 
acceptable threshold to make reliable decisions based 
on observations from individuals. Coefficients above 
.70 are ideal and provide stronger evidence of 
reliability. The results of the analyses for independent 
and group reviews are presented in the Study Process 
Findings subsection in the Results section. 


RESULTS 


The results from the CCCA study provide a wealth of 
information regarding the knowledge, skills, and 
abilities that are considered prerequisite for entry into 
credit-bearing college courses without remediation. 
This section of the report consists of two major 
subsections. Presented first are data regarding the 
evidence for reliability and validity associated with the 
implementation of the processes articulated in the 
study’s design. Second are the findings regarding the 
knowledge, skills, and abilities found to be 
prerequisite during the review process. 

Study Process Findings 

The reliability of coding, and the internal and external 
validity of findings based on study processes are vital 
elements to consider prior to an examination of the 
prerequisite KSAs. As described in the Methods 
sections, the study incorporated a number of 
processes to maximize the probability of obtaining 
reliable and valid results. The results from the 


measures taken to maximize and monitor reliability 
and validity are described below. 

Institutional Representativeness for Operational 
Course Packets 

To determine representativeness, only the 151 
institutions from which operational course packets 
were drawn were included in the analyses. Only 
operational course packets were used to determine 
institutional representativeness because they were the 
packets from which data were summarized into 
content maps of prerequisite KSAs; operational 
course packet data also were the data used by the 
NAEP experts to construct the narrative descriptions. 
The 151 institutions that submitted course artifacts 
that were used in the operational course packets were 
analyzed for their representativeness in relation to the 
overall population of institutions by applying the four 
representativeness criteria used to stratify the sample. 
Table 13 presents the percentages by institutional 


Table I 3. Expected and Observed Percentages, by Institutional Characteristic, in the Sample of 77 Institutions Providing 
Artifacts for the Operational Course Packets Used in Reviews of Courses With Substantial Reading Demands 



Small enrollment % 

Medium enrollment % 

Large enrollment % 

Type, region, and control 

Expected Observed 

Expected Observed 

Expected Observed 


2-year 


4-year 


East 


West 


East 


West 


Public 

Private 

Public 

Private 

Public 

Private 

Public 

Private 


8 

2 

8 

I 

4 

20 

3 

17 


3 

I 

8 

I 

3 

21 

5 

12 


3 

0 

5 

0 

3 

2 

2 

2 


0 

0 

4 
0 
8 
0 
3 

5 


2 

0 

5 

0 

4 
I 

5 

I 


0 

0 

8 

0 

4 

3 

13 

0 


Note. Due to rounding, percentages may not sum to 100%. 






characteristic of the population and sample for those 
institutions providing packets used as operational 
packets for the courses with substantial reading 
demands (i.e., English literature, psychology, U.S. 
government, and U.S. history). 

Table 14 presents similar information for the 
mathematics operational packet sample. 

Chi-square (x 2 ) analyses compared the 
representativeness of the CCCA sample of institutions 
to the U.S. population of institutions along the four 
representativeness criteria. The expected data come 
from the population of 3,010 institutions and the 
observed data from the operational course packet 
sample of 151 institutions. Eight separate X analyses 
(one for each of the four representativeness criteria in 
both mathematics and reading) compared the observed 
proportion of institutions in the CCCA sample to the 
expected proportion of institutions in the U.S. 
population of institutions. 

Institutional representativeness for course packets 
from courses with substantial reading demands. The 

X 2 analyses, based on an a priori alpha level of 0.05, 


show no significant differences by institutional 
characteristic in the institutions providing operational 
packets reviewed by the reading content experts. This 
analysis indicates that the reviewed artifacts, compiled 
into course packets, represent a sample of institutions 
that is representative of the population of institutions 
(see Table 15). 

Institutional representativeness for mathematics 
course packets. The X analyses on the 
representativeness of the institutions that provided 
mathematics artifacts used in the operational course 
packets indicate that the artifacts underrepresent small 
institutions and private institutions and overrepresent 
public institutions and large institutions (see Table 16). 

The relationship between the representativeness of 
the mathematics sample and the external validity of 
the findings was investigated using three sets of 
analyses. First, post hoc X analyses were performed 
on the mathematics sample, filtering by size. A 
second set of analyses artificially reduced the sample 
to one that was representative in all institutional 
characteristics and compared the two samples to 
provide insight into degree to which the sample’s 


Table 14. Expected and Observed Percentages, by Institutional Characteristic, in the Sample of 74 Institutions Providing 
Artifacts for the Operational Course Packets Used in Reviews of Mathematics Courses 


Small enrollment % 

Medium enrollment % 

Large enrollment % 

Type, region, and control Expected Observed 

Expected Observed 

Expected Observed 

Public 8 7 

East 

Private 2 1 

3 1 

0 0 

2 3 

0 0 

2-year 

Public 8 8 

West 

Private 1 0 

5 4 

0 0 

5 8 

0 0 

Public 4 3 

East 

Private 20 8 

3 5 

2 5 

4 1 1 

1 0 

4-year 

Public 3 3 

West 

Private 17 15 

2 1 

2 3 

5 12 

1 1 


Note. Due to rounding, percentages may not sum to 100%. 




Table 15. Representativeness of the 77 Institutions Submitting Artifacts Used in the Operational Course Packets for Courses 
With Substantial Reading Demands 


Institutional 


Expected number 

Observed number 


characteristic 

Category 

of institutions 

of institutions 

2 

X 


2-year 

27 

19 


Program type 



3.61 


4-year 

50 

58 



Small 

49 

41 


Enrollment size 

Medium 

14 

15 

4.49 


Large 

14 

21 



East 

39 

32 


Geographic region 

West 

39 

45 

2.20 


Public 

40 

44 


Control of institution 

Private 

37 

33 

0.67 


characteristics affected the NAEP experts’ 
construction of the narrative descriptions based on 
prerequisite KSAs and non-NAEP additional KSAs. 
Finally, the researchers examined the number of 
average student populations of the institutions by 
characteristics to estimate the student-level impact of 
the underrepresentation of institutions with certain 
characteristics. All follow-up analyses and their results 
are presented below. 


Post hoc y 2 analyses. A series of yj analyses were 
performed on the mathematics courses, after filtering 
by size, to determine the effect of the sample 
characteristics on generalizability of study findings. 
Table 17 summarizes these results. After filtering by 
size, the y~ analyses are not significant for any other 
characteristic, meaning that no one characteristic 
other than size was significantly contributing to 
deviation from expected values. 


Table 16. Representativeness of the 74 Institutions Submitting Artifacts Used in the Operational Course Packets for 


Mathematics Courses 

Institutional 


Expected number 

Observed number 


characteristic 

Category 

of institutions 

of institutions 

2 

X 


2-year 

26 

24 


Program type 



0.21 


4-year 

48 

50 



Small 

47 

33 


Enrollment size 

Medium 

13 

15 

1 5. 1 8*** 


Large 

14 

26 



East 

37 

33 


Geographic region 

West 

37 

41 

0.87 


Public 

39 

49 


Control of institution 

Private 

36 

25 

5.58* 


Note. *p < .05. ***p < .001. 






Table I 7. Comparisons of Expected to Observed Number of Institutions Contributing Operational Course Packets for 


Mathematics Courses, Filtered by Institutional Control 


Institutional 


Expected number 

Observed number 


characteristic 

Category 

of institutions 

of institutions 

X 

Small schools (n = 33) 


2-year 

10 

12 


Program type 

23 


0.57 


4-year 

21 



East 

17 

14 


Geographic region 

West 

16 

19 

1.09 


Public 

12 

15 


Control of Institution 

Private 

21 

18 

1.18 

Medium schools (n = 15) 


2-year 

8 

4 


Program type 

8 

1 1 

2.41 


4-year 



East 

7 

9 


Geographic region 

West 

8 

6 

1.07 


Public 

12 

9 


Control of Institution 

Private 

3 

6 

3.75 

Large schools (n = 26) 


2-year 

10 

8 


Program type 

16 

18 

0.65 


4-year 



East 

1 1 

10 


Geographic region 

West 

15 

16 

0.16 


Public 

23 

25 


Control of institution 

Private 

3 

1 

1.51 


Another set of y 2 post hoc analyses was conducted, 
filtering by public or private control. Table 18 
summarizes these results. The y 2 analysis indicated 
that public institutions were not representative of the 
population in terms of program type and size because 
two-year public and small- and medium-sized public 
institutions were underrepresented. The y 2 analysis 
for the private institutions indicated that they were 
not fully representative of the population because 
small private institutions were underrepresented. 


Comparison of findings to reduced representative 
sample. As a second exploration of the potential effects 
of the representativeness of the mathematics sample, die 
research team randomly deleted nine institutions from 
the oversampled institutional categories to create an 
artificial sample that would have been precisely 
representative of the population. Analysis determined 
that the revised sample, which included 17-18 packets 
per course title rather than 20 packets per course tide, 
would have produced reliable findings for this study, 
according to the modified decision (D) study that was 






Table 1 8. Comparisons of Expected to Observed Number of Institutions Contributing Operational Course Packets for 
Mathematics Courses, Filtered by Institutional Control 


Institutional 


Expected number 

Observed number 


characteristic 

Category 

of institutions 

of institutions 

X 

Public schools (n = 49) 


2-year 

30 

23 

4.21* 

Program type 

19 

26 


4-year 



Small 

21 

15 


Enrollment size 

Medium 

13 

9 

8.92* 


Large 

15 

25 



East 

22 

22 


Geographic region 

West 

27 

27 

0.00 

Private schools (n = 25) 


2-year 

2 

1 


Program type 



0.54 


4-year 

23 

24 



Small 

22 

18 


Enrollment size 

Medium 

2 

6 

8.73* 


Large 

1 

1 



Small 

14 

1 1 


Geographic region 

West 

1 1 

14 

1.46 


Note. *p < .05. 


performed. 10 The researchers examined the similarities 
and differences in die distribution of prerequisite KSAs 
between the sample of 74 institutions and the revised 
sample of 65 institutions. Specifically, three institutions 
that contributed statistics packets and two each 
contributing packets for finite mathematics, 
precalculus/ calculus, and college algebra were randomly 
excluded from this analysis. The resulting sample of 65 
institutions (71 course packets) was then representative 
across all four institutional characteristics. 


10 The results of the modified D-study using 17 courses packets produced results 
that were consistent with the D-study using 20 course packets. The average 
difference between the eight coefficients generated during both D-studies was 
only .63%, and every coefficient was above the acceptable threshold of .70 
(coefficients were between .78— .92). This suggests that using a sample size of 17 
courses per course title would have only a slight negative effect on the reliability 
of the findings. See Generalizability Study section for more information on the 
method. 


The research team compared the content maps of 
prerequisite KSAs from the full sample with the 
content maps from the revised sample using the same 
80% threshold of group agreement that the 
mathematics experts used to make decisions on when 
to include an objective within the narrative 
descriptions. The 80% level of agreement is defined 
as the four content review groups finding any given 
objective to be prerequisite or prerequisite and 
important in 80% of the course packets within a 
course title. Results from this analysis resulted in 
minimal changes in the content maps of the revised 
sample from the content maps of the full sample. 

This suggests that the NAEP experts would have 
received data that was very similar in distribution of 
prerequisite KSAs. The resulting conclusions reached 
by the NAEP experts would have likely been the 





same or similar to those reached based on their 
review of the full sample. 

Analysis of student populations. The third exploration 
of the potential effects of the representativeness of the 
mathematics sample used student population data 
from 2011 to estimate the percentage of students 
enrolled at the institutions in the CCCA sample. First, 
the representativeness profile of the sample was 
established by reviewing Table 16. This review 
identified four types of institutions in which the 
expected and observed percentage of institutions 
differed by at least three percentage points. Of these 
four, small private 4-year institutions in the eastern 
U.S. were the only type of institution with substantial 
underrepresentation in the CCCA sample. Whereas 
20% of the sample would have been expected to be 
from these types of institutions, the sample contained 
8%. However, small private 4-year institutions in the 
eastern U.S. enroll only 4% of the 24,859,900 
postsecondary students in the 3,010 institutions that 
comprise the population for this study. In other words, 
the eastern small private 4-year institutions category 
would have been expected to represent just less than 
one million students but the sample in fact represents 
just under 400,000 students. 

The representativeness of three other three types of 
institutions in the CCCA sample was also examined. 
These were large public 4-year institutions in the 
eastern U.S. (4% expected and 11% observed), large 
public 4-year institutions in the western U.S. (5% 
expected and 12% observed), and public 2-year 
institutions in the western U.S. (5% expected and 8% 
observed). These institutions collectively enrolled 
48% of all students within the population of 3,010 
institutions in 2011. Overrepresentation of these three 
institution types, then, reflected a collective set of 
institutions with larger student enrollment, which 
ultimately relates the CCCA study’s inferences to a 
larger set of students. 


Conclusion. The results of the follow-up analyses on 
the effects of sample representativeness by institution 
type for mathematics strongly suggest that deviations 
from representativeness in the four identified areas do 
not pose a serious threat to validity. It is important to 
note that the study is not intended to apply its 
findings to specific institution types from among the 
entire sample. Instead, it is designed to reach overall 
judgments about the presence of prerequisite KSAs 
among the full set of mathematics courses included in 
the sample. Furthermore, the results of these analyses 
find no reason to conclude that the substantive 
findings are systematically biased in one direction or 
another with regard to the determination of 
prerequisite KSAs and the NAEP mathematics 
experts’ development of the narrative description of 
those KSAs a student needs to be prepared to enter a 
credit-bearing college course. 

Artifact Sufficiency 

The sufficiency of the artifacts to allow for consistent 
determinations of prerequisite KSAs is one of the 
three key elements of the method that contribute to 
the validity of the study: institutional 
representativeness, artifact quality/ sufficiency, and 
reviewer reliability. Artifacts as they were grouped 
into packets were sufficient if they represented the 
range of materials that would be found in these entry- 
level courses across instructors and institutions; in 
other words, the range of artifacts, as compiled into 
course packets, were sufficient if they achieved the 
principal of redundancy. As shown in the content 
maps of prerequisite KSAs, the course packets 
allowed for a range of codings within course titles. 

Artifacts also are sufficient for the purposes of this 
study if they allowed reviewers to focus on (1) what 
students are expected to be able to learn, via analyses 
of course goals and objectives; (2) what students are 
expected to be able to do to demonstrate they have 
learned, via analyses of assessments and assignments; 


and (3) the kinds of content the students will reference 
in order to learn, via analyses of course texts. The 
design of the study incorporated multiple processes to 
ensure that each packet contained artifacts to allow for 
all three types of analyses by reviewers. 

During the NAEP expert advisory panels preceding 
the training of reviewers, the NAEP experts vetted 
the protocols for the type, number, and quality of the 
artifacts. The Governing Board reviewed and 
approved the protocols prior to their implementation. 
Almost all packets consisted of three to five artifacts; 
the numbers were kept similar to constrain the 
variance attributable to number of artifacts. As 
materials were submitted and reviewed, the research 
team assessed each packet to ensure that, as a whole, 
the group of artifacts comprised within a packet had 
sufficient data richness to permit identification of 
prerequisite KSAs. To explore whether the number of 


artifacts influenced the number of KSAs that were 
identified, the research team divided the operational 
course packets into four groups by number of 
artifacts. See Tables 19 and 20 for findings. 

Collapsing across course titles with substantial reading 
demands, researchers conducted an analysis of 
variance with number of artifacts as the independent 
variable and number of identified prerequisite KSAs 
as the dependent variable. The results were not 
significant, F( 3, 76) = 2.41 ,p > 0.05, suggesting that 
the number of artifacts in a packet did not 
significantly influence the identification of KSAs. 

A similar analysis of variance was conducted on the 
combined mathematics packets. The results were not 
significant, F( 3, 73) = 0.11,p > 0.05. 


Table 1 9. For the Courses With Substantial Reading Demands, the Average Number of NAEP-Specific KSAs Identified in 
Course Packets Within Each Course Title by Number of Artifacts 


Course title 

3 artifacts 

4 artifacts 

5 artifacts 

> 5 artifacts 

Number 
of packets 

Number 
of KSAs 

Number 
of packets 

Number 
of KSAs 

Number 
of packets 

Number 
of KSAs 

Number 
of packets 

Number 
of KSAs 

English literature 

12 

22 

4 

19 

2 

25 

2 

25 

Psychology 

13 

17 

1 

22 

6 

21 

0 

0 

U.S. government 

1 1 

19 

5 

17 

3 

20 

1 

23 

U.S. history 

9 

18 

6 

23 

2 

23 

3 

22 


Table 20. For Mathematics Courses, the Average Number of NAEP-Specific KSAs Identified in Course Packets Within Each 


Course Title by Number of Artifacts 



3 artifacts 

4 artifacts 

5 artifacts 

> 5 artifacts 


Number 

Number 

Number 

Number 

Number 

Number 

Number 

Number 

Course title 

of packets 

of KSAs 

of packets 

of KSAs 

of packets 

of KSAs 

of packets 

of KSAs 

Precalculus/calculus 

14 

28 

3 

25 

3 

23 

0 

0 

College algebra 

13 

19 

4 

19 

3 

19 

0 

0 

Finite mathematics 

13 

18 

5 

20 

1 

20 

1 

23 

Statistics 

12 

18 

7 

19 

1 

22 

0 

0 






Content reviewers were queried at several points 
about the artifacts and about their ability to make 
decisions about the perquisite KSAs given the study’s 
methods. To determine the usefulness or helpfulness 
of each type of artifact to the review process, the 
content reviewers coded the helpfulness of each 
source of evidence provided in the course packet. In 
studies incorporating document analyses, effective 
evidence sources are expected to be helpful to 
reviewers in meeting study objectives. 

In mathematics course packets, content reviewers found 
the course syllabi and textbook excerpts to be helpful 
sources of evidence; 86% of textbook excerpts were 
coded as “very helpful” or “helpful” and 52% of syllabi 
were coded as such. Assignments 11 were coded “not 
applicable” in 60% of mathematics course packets, and 
assessments were coded as “not applicable” in 43% of 
mathematics course packets. A code of “not applicable” 
indicates that the artifact was not present. In the 
mathematics course packets where assignments were 
applicable, 50% were coded as very helpful or helpful. 

In the mathematics course packets where assessments 
were applicable, 39% were coded as very helpful or 
helpful. The ratings suggest that the textbook excerpt 
was the most valuable artifact for mathematics reviewers 
to have, and that die assignments from the textbook 
were more helpful than assignments or assessments 
independent of the textbook. 

In course packets from courses with substantial reading 
demands, content reviewers found the course syllabi and 
assignments to be helpful sources of evidence; 94% of 
syllabi were coded as “very helpful” or “helpful.” 
Reviewers coded text excerpts as “very helpful” or 
“helpful” in 42% of the course packets reviewed for 
prerequisite KSAs related to reading. In the course 
packets reviewed for reading content where an 


11 For course packets where the assignment or assessment was a problem set 
within the textbook excerpt, helpfulness ratings were applied only to the 
textbook, and the assignment or assessment was coded as “not applicable.” 


assignment was applicable to the course packet, 90% 
were coded as “helpful” or “very helpful.” In the course 
packets where assessments were applicable, 66% were 
coded as “helpful” or “very helpful.” Appendix T 
provides more detail on ratings of artifact helpfulness. 

Using the artifacts within the packets, the content 
reviewers identified prerequisite KSAs, including 
NAEP-specific KSAs and non-NAEP additional KSAs. 
The reviewers were asked several questions during the 
process evaluation. All reviewers agreed with the 
statement, “Overall, I am satisfied with how I coded my 
packets during independent review.” After the group 
review, satisfaction with the process was very high; for 
instance, 92% of mathematics and 100% of reading 
reviewers agreed with the item, “I am confident in the 
outcomes of the group review process.” See the Process 
Evaluations section for more information. 

In the larger question of whether the number of 
artifacts that were comprised within each packet was 
sufficient for reviewers to validly identify prerequisite 
KSAs, there is no evidence to suggest that any 
particular characteristic of the sample of artifacts (e.g., 
number, type, content, specificity, etc.) would have 
influenced coding in a systematic way. The total 
variance attributable to the interaction of KSAs by 
packets is 15% in mathematics and 27% in reading; 
much of this variance would be accounted for by the 
to-be-expected differences among course titles (see 
Generalizability Study Results later in this section). 
Some variance might be attributable to qualities 
within the packet, but it is not possible given the 
current data to know how much would be attributable 
to any given packet characteristic. It is clear, however, 
from the analyses of variance, that the number of 
artifacts did not significantly affect the number of 
NAEP-specific KSAs that reviewers identified. 

The artifacts represented a range of materials students 
would encounter at the beginning of a course. The 


internal logic of the methods selected for this study 
suppose that early materials provide the best source 
of evidence regarding the prerequisite KSAs because 
later materials are likely to be building on the KSAs 
students are developing as a result of being in the 
course. These are not prerequisite KSAs. The only 
title within the eight selected for this study for which 
this logic may hold less strongly is finite mathematics 
because the discrete topics selected to be taught in 
this course can differ in sequence and the topics can 
be minimally related to one another. 

Levels of Reviewer Agreement 
The extent to which reviewers agree with each other is 
key to internal reliability of study findings. Reviewer 
agreement is defined as the three reviewers in each 
group independently identifying (1) the applicability of 
the objective as a prerequisite KSA to the course under 
review, and (2) the same KSA exclusion(s) for each 
prerequisite KSA. If one or more reviewers differed on 
either decision, these counted as a disagreement. 

Because this study included a multiphased design with 
multiple discrete outcomes at every decision point, 
there are several ways to present the levels of reviewer 
agreement. Reviewers could agree during independent 
review, disagree during independent review but come 
to consensus during group review, or disagree during 
independent review and not reach consensus 
regarding the applicability of an objective as 
prerequisite. The reviewers also could agree or 
disagree at both independent and group review on the 
applicability of an objective as a prerequisite KSA 
(applicability code) and whether an objective that was 
not considered a prerequisite KSA as written would 
become a prerequisite KSA if a part of that objective 
was excluded (exclusion code). 

Independent review agreement. The interim 
independent decisions in a convergent consensus 
content review approach are expected to reflect each 


individual’s expertise (see Appendix U for content 
reviewer characteristics). Some discrepancies in coding 
are expected prior to group discussion and the 
convergence into a single decision representing the 
collective expertise of individuals in the group. As 
such, higher or lower independent review agreement 
does not necessarily indicate more or less reliability in 
the review process or outcomes. Rather, higher or 
lower independent review agreement is simply the first 
step toward building a group decision. 

Table 21 describes the similarity of individual 
decisions made by the 12 reading and 12 mathematics 
content reviewers during independent review. During 
independent review, overall agreement in 
mathematics courses reached 86%; in the 14% of 
decision points where there was disagreement, 1 2% 
were disagreements regarding applicability and 2% 
were disagreements regarding an exclusion. Overall 
agreement in courses with substantial reading 
demands reached 56%; among the 44% that were 
points of disagreement, 30% were disagreements 
regarding applicability and 14% related to exclusions. 
The independent review data are interim data points 
within the convergent consensus process. 

Group review agreement. During group review, 
individuals convened in groups of three to discuss 
each decision that was not unanimous in the 
independent review, to share evidence, and to make 
decisions determining whether each NAEP objective 
is prerequisite and if there are any KSA exclusions for 
each NAEP objective in each course. Table 22 
provides the rate of agreement for these decisions. 

Note that the total number of group decisions (first 
column) is the same as the number of agreements 
subtracted from the total number of decisions. This 
provides a level of agreement percentage 
corresponding to interrater reliability. 


Table 21. Independent Review Agreement — Prerequisite KSAs and KSA Exclusions 


Course title Total number of decisions 

Number of agreements 

Agreement % 

Mathematics total (n = 112) 

29,120 

24,952 

86 

Precalculus/calculus 

7,280 

5,790 

80 

College algebra 

7,280 

6,070 

83 

Finite mathematics 

7,280 

6,425 

88 

Statistics 

7,280 

6,667 

92 

Reading total (n = 112) 

8,288 

4,675 

56 

English literature 

2,072 

1,024 

49 

Psychology 

2,072 

1,270 

61 

U.S. government 

2,072 

1,184 

57 

U.S. history 

2,072 

1,197 

58 

Note. Table includes the decisions for the precalculus/calculus course packet that was later removed from the study. The 1 1 2 
course packets overall in mathematics and reading refers to the number of course packet reviews completed at group review. 


Each operational course packet was reviewed by one content review group (80 x I = 80). Each of the validation course packets 
was reviewed by each of the content review groups (8x4 = 32). Therefore, the number of mathematics course packet reviews 
overall is 112, and the number of reviews of packets for courses with substantial reading demands is 112. 


Table 22. Group Review Agreement — Prerequisite KSAs and KSA Exclusions 


Course title 

Total number of group 
decisions (disagreements 
during independent review) 

Agreement after group 
review % 

No group decision after 
group review (agree to 
disagree) % 

Mathematics total 

4,168 

96 

4 

Precalculus/calculus 

1,490 

96 

4 

College algebra 

1,210 

98 

2 

Finite mathematics 

855 

98 

2 

Statistics 

613 

95 

5 

Reading total 

3,613 

97 

3 

English literature 

1,048 

94 

6 

Psychology 

802 

99 

1 

U.S. government 

888 

96 

4 

U.S. history 

875 

100 

0 


Note. Table includes the decisions for the precalculus/calculus course packet that was later removed from the study. 


The level of agreement was high across all eight 
course titles. In both mathematics and reading, 
content review groups agreed at least 94% of the time 
across all possible decisions. Agreement did not differ 
when the first 10 courses reviewed were compared to 


the last 10 courses reviewed, suggesting an absence of 
order effects. 

Reasons for changes in coding decisions. When 
reviewers changed their independent decisions in group 






review, scribes recorded the reasons. See Appendix P 
for information on scribe training, including applicability 
of reasons for changes in coding decisions. 

Tables 23 and 24 provide the percentages of response 
changes resulting from each reason. 

In mathematics, the most frequent reason for selecting 
“other” was due to rescinding or agreeing to new KSA 
exclusions (e.g., “added exclusion” and “removed 
exclusion”). The “other” selection was also used to 
specify when the reviewer changed his or her mind due 
to importance (e.g., “new understanding of importance” 
and “compromised to prerequisite rather than 
important”). Mathematics reviewers also selected 
“other” when they had a “new understanding of [the] 
nature of [the] course” or had “previous discussion” 
about the nature of the course and decision rules. 

In courses with substantial reading demands, the most 
frequent reason provided for “other” was the group 
member changing his or her mind regarding a KSA 

Table 23. Reasons Content Reviewers Indicated for 
Changing Independent Coding During Group Review in 
Course Packets From Courses With Substantial Reading 
Demands 


Reason % 

Evidence not seen before, changed 
understanding of evidence, or disputed 
evidence 

37 

New shared understanding of the objective 

17 

Other 

16 

Changed understanding of terminology 

14 

Other group member(s) made convincing 
argument 

1 1 

Changed understanding of application of 
decision rule 

2 

Technical error with survey tool 

2 


Note. Disputed evidence indicates a content reviewer 
agreed to change independent coding at group review based 
on other content reviewers in the group challenging the 
sufficiency of the evidence cited. 


exclusion. For example, “No exclusion” and “Yes 
exclusion” were indications of whether the reviewer 
rescinded a KSA exclusion or agreed to a new KSA 
exclusion, respectively. The “other” selection was also 
used when reviewers wanted to specify which decision 
rule caused them to change their mind (e.g., the 
decision mle that guides reviewers on how to translate 
the word ‘or’ in the NAEP framework statements). 

Consistent with the convergent consensus process, the 
most common reason cited for an individual changing 
his or her code on a KSA exclusion or prerequisite 
KSA during group review was due to consideration of 
evidence seen or described by others. This reason 
resolved 48% of the mathematics disagreements and 
37% of the reading disagreements. Reviewers also 
changed their minds to reflect the shared 
understanding that emerged through group discussion 
(20% among mathematics disagreements; 17% among 
reading). Validity of the resulting findings is enhanced 
by this group review of the evidence against the 
decision mles (Bressen, 2007; Dalkey, 1969). 


Table 24. Reasons Content Review Indicated for Changing 
Independent Coding During Group Review in Course 
Packets From Mathematics Courses 


Reason % 

Evidence not seen before, changed 
understanding of evidence, or disputed 
evidence 

48 

New shared understanding of the objective 

20 

Other group member(s) made convincing 
argument 

12 

Other 

1 1 

Changed understanding of application of 
decision rule 

5 

Technical error with survey tool 

Changed understanding of terminology 

3 

1 


Note. Disputed evidence indicates a content reviewer 
agreed to change independent coding at group review 
based on other content reviewers in the group challenging 
the sufficiency of the evidence cited. 






A threat to the validity of the convergent consensus 
approach is that during group discussions, one 
member of the group may dominate the discussion or 
aggressively advance a position; conversely, a member 
may be extremely passive and agree with everything 
others suggest (Bressen, 2007). To determine the 
presence and impact of this potential “bully effect,” the 
research team analyzed how many times each group 
member changed their independent review coding and 
the rationale for each change. Although it is expected 
that individual responses may change during the group 
review process, substantial differences in change rates 
across reviewers in a group, especially when there are 
very few changes for an individual group member, may 
suggest the presence of a “bully effect.” Tables 25 and 
26 provide the percentages of response changes 
attributable to the members within each group. 


The results show that individual member’s changes in 
coding were spread fairly equally across members, 
which suggests no member had more influence than 
another. The findings suggest that group reviews were 
collaborative, that debates and response changes were 
based on evidence and new shared understandings, 
and that personality and social factors had little 
impact on group outcomes. 

Appendix V provides rationales for every response 
change for every reviewer. 

Generalizability Study Results 
A generalizability study was conducted to establish 
estimates regarding the interrater and intrarater 
reliability of the content reviewers during both the 
independent and group review phases. The 


Table 25. Response Changes Attributable to Each 
Mathematics Group Member 






% of 

Group mem 

ber 



changes 

Who in Group 1 (n = 

1,930) 

changed 

their 

mind? 

Member One 




23 

Member Two 




51 

Member Three 




24 

Who in Group 2 (n - 

- 863) changed their i 

mind? 

Member One 




26 

Member Two 




37 

Member Three 




37 

Who in Group 3 (n = 

1,249) 

changed 

their 

mind? 

Member One 




32 

Member Two 




37 

Member Three 




31 

Who in Group 4 (n = 

U28) 

changed 

their 

mind? 

Member One 




35 

Member Two 




37 

Member Three 




29 


Note. Due to rounding, percentages within a single group 


may not sum to 100%. 


Table 26. Response Changes Attributable to Each Reading 
Group Member 


% of 

Group member changes 


Who in Group I (n = 1,178) changed their mind? 

Member One 26 

Member Two 40 

Member Three 34 

Who in Group 2 (n - 1,037) changed their mind? 

Member One 35 

Member Two 44 

Member Three 22 

Who in Group 3 (n = 909) changed their mind? 

Member One 24 

Member Two 40 

Member Three 37 

Who in Group 4 (n = 945) changed their mind? 

Member One 19 

Member Two 34 

Member Three 47 

Note. Due to rounding, percentages within a single group 
may not sum to 100%. 




Table 27. Variance Components Estimation for Mathematics Content Reviewer Judgments — Two-Facet Fully Crossed Design 
KSA Objective-Level Analysis 


Generalizability Study 
Variance component(s) 

Variance 

estimation 

Variance 

estimation 

% 

Number 

Absolute 

error 

Relative 

error 

P 

<t> 

KSA Objective (KSA) 

0.063 

24.61 

130 

0.010 

0.008 

0.884 

0.858 

Course Packet 

0.012 

4.69 

8 





Reviewer 

0.007 

2.73 

12 





KSA x Course Packet 

0.038 

14.84 






KSA x Reviewer 

0.030 

1 1.72 






Course Packet x 

Reviewer 

0.007 

2.73 






Course Packet x 

Reviewer x KSA 

0.099 

38.67 






D-study 






0.823 

0.791 


Note. How to understand the variance components: KSA objective variance refers to the extent to which the KSAs differ from 
one another in the frequency with which they are considered prerequisite; course packet variance refers to some packets having 
higher numbers of prerequisite KSAs than other packets; reviewer variance examines the extent to which individual reviewers 
differed from each other in their overall pattern of coding KSAs. The interaction terms can be understood as follows: The 
coding of the KSA x Course Packet interaction is a measure of the extent to which the determination of whether a KSA is 
considered prerequisite is a function of the course packet being reviewed; the KSA x Reviewer interaction is a measure of the 
extent to which the determination of whether a KSA is considered a prerequisite is a function of the reviewer; the Course 
Packet x Reviewer interaction considers whether reviewers provide different coding in some packets versus others; and the 
Course Packet x Reviewer x KSA is additional random variation not attributable to any particular characteristic that was studied. 


generalizability study was designed and conducted 
using a set of validation course packets coded during 
the independent and group reviews. A decision study 
investigated whether the levels of reliability were 
maintained in the reviews of operational course 
packets. This section presents the findings from the 
generalizability and decision studies. 

Independent review. For die independent review, the 
results indicate that the reviewers were quite consistent 
and reliable in coding the validation course packets. The 
relative g-coefficients (p) for die 12 reviewers coding the 
KSAs in the eight validation course packets were high: 
.853 for courses with substantial reading demands and 
.884 for mathematics. The absolute g-coefficients ((j)) 
were .833 and .858 for reading and mathematics, 
respectively. This indicates high reliability and 
consistency among the reviewers. In addition, the D- 
study estimates for coding of operational course packets 


indicate that these high levels of reliability were 
maintained when the reviewers coded the operational 
course packets. The (p) and (())) coefficients for 
matiiematics were .823 and .791; for courses with 
substantial reading demands they were .854 and .809. 
Thus, the analyses indicate that the independent reviews 
possess high levels of reliability in both the validation 
course packets and in operational course packets. 

Tables 27 and 28 summarize the results of the fully 
crossed analyses 12 of independent ratings at the level 
of the NAEP framework objectives. 


12 A generalizability analysis is fully crossed when all content reviewers code the 
same course packets using the same framework. In terms of the CCCA 
Independent Review Generalizability Study, all 1 2 content reviewers in each 
subject area reviewed the same eight course packets using the NAEP framework. 
In the CCCA Group Review Generalizability Study, all four groups in each 
subject area reviewed the same eight course packets using the NAEP framework. 




Table 28. Variance Components Estimation for Reading Content Reviewer Judgments — Two-Facet Fully Crossed Design KSA 
Objective-Level Analysis 


Generalizability Study 
Variance component(s) 

Variance 

estimation 

Variance 

estimation 

% 

Number 

Absolute 

error 

Relative 

error 

P 

<t> 

KSA Objective (KSA) 

0.144 

25.17 

37 

0.029 

0.025 

0.853 

0.833 

Course Packet 

0.014 

2.45 

8 





Reviewer 

0.025 

4.37 

12 





KSA x Course Packet 

0.157 

27.45 






KSA x Reviewer 

0.042 

7.34 






Course Packet x 

Reviewer 

0.021 

3.67 






Course Packet x 

Reviewer x KSA 

0.169 

29.55 






D-study 






0.854 

0.809 


The independent review subtopic (mathematics) or 
standard (reading) level coding showed comparable 
consistency. Tables 29 and 30 provide the g-coefficients 
for mathematics and reading for the subtopic/ standard 
ratings, which are .924 and .897 for (p) and (())) 
coefficients for mathematics, and .899 and .885 for (p) 
and (())) coefficients for reading. The D-study 
coefficients likewise indicate that the independent 
review operational coding process of using three 


content reviewers to code 20 course packets was reliable 
at both the objective and subtopic/ standard level with 
values of .872 (p) and .831 ((j)) for mathematics and .893 
(p) and .841 (()>) for reading. 

Tables 29 and 30 summarize the results of the fully 
crossed analysis of independent ratings at the level of 
the NAEP framework subtopics or standards. 


Table 29. Variance Components Estimation for Mathematics Content Reviewer Judgments — Two-Facet Fully Crossed Design 
Subtopic-Level Analysis 


Generalizability Study 
Variance component(s) 

Variance 

estimation 

Variance 

estimation 

% 

Number 

Absolute 

error 

Relative 

error 

P 

<t> 

Subtopic 

0.161 

31.94 

25 

0.018 

0.013 

0.924 

0.897 

Course Packet 

0.025 

4.96 

8 





Reviewer 

0.023 

4.56 

12 





Subtopic x Course 

Packet 

0.056 

ll.l 1 






Subtopic x Reviewer 

0.054 

10.71 






Course Packet x 

Reviewer 

0.012 

2.38 






Course Packet x 

Reviewer x Subtopic 

0.173 

34.33 







D-study 


0.872 


0.831 






Table 30. Variance Components Estimation for Reading Content Reviewer Judgments — Two-Facet Fully Crossed Design 
Standard-Level Analysis 




Variance 






Generalizability Study 

Variance 

estimation 


Absolute 

Relative 



Variance component(s) 

estimation 

% 

Number 

error 

error 

P 

<l> 

Standard 

0.214 

34.68 

10 

0.028 

0.024 

0.899 

0.885 

Course Packet 

0.000 

0.00 

8 





Reviewer 

0.044 

7.13 

12 





Standard x Course 

Packet 

0.148 

23.99 






Standard x Reviewer 

0.047 

7.62 






Course Packet x 

Reviewer 

0.015 

2.43 






Course Packet x 

Reviewer x Standard 

0.149 

24.15 






D-study 






0.893 

0.841 


Group review. Similar to the independent review 
results, the group reviews show consistency and 
reliability in the sample of validation course packets, 
with all coefficients estimated in excess of .80. 

Reading g-coefficient estimates were consistent as 
well. Reliability estimates for group coding of 
objectives among the packets for courses with 
substantial reading demands were .838 (p) and .804 
(cj>); for group coding at the level of standards, they 
were .887 (p) and .856 ((J>). 

The mathematics g-coefficients approached or 
exceeded .90 for both objective and subtopic levels in 
group coding. Reliability estimates for group coding 
of objectives among the mathematics packets were 
.880 (p) and .873 (())); for group coding at the level of 
standards, they were .926 (p) and .897 (([)). 

Tables 31 and 32 provide the results of the analyses 
performed on the group judgments at the level of the 
NAEP framework objectives. Tables 33 and 34 


provide the results using the NAEP framework 
subtopics or standards as the unit of analysis. 

The D-study estimates suggest that the operational 
coding process of having a single group code 20 course 
packets yielded high reliability for objective coding 
[.880 (p) and .873 (([))] and subtopic coding [.926 (p) 
and .897 (4>)] in mathematics. The D-study estimates 
for the group coding of objectives [.808 (p) and .720 
(4>)] and standards [.868 (p) and .765 (([))] were lower in 
reading than comparable coefficient estimates for 
coding in mathematics, but they nonetheless continue 
to demonstrate acceptable (> .70) levels of consistency 
and reliability of coding in operational practice. 

In summary, this generalizability study provides 
evidence of the reliable and consistent coding of 
objectives and subtopics (mathematics) or standards 
(reading) within course packets during both 
independent and group review. Of all the various 
generalizability coefficients estimated, every one of 
them exceeded an acceptable threshold of .70; more 




Table 3 I . Variance Components Estimation for Mathematics Group Judgments — Two-Facet Fully Crossed Design KSA 
Objective-Level Analysis 


Generalizability Study 
Variance component(s) 

Variance 

estimation 

Variance 

estimation 

% 

Number 

Absolute 

error 

Relative 

error 

P 

<t> 

KSA Objective (KSA) 

0.133 

38.22 

130 

0.017 

0.014 

0.904 

0.889 

Course Packet 

0.020 

5.75 

8 





Group 

0.000 

0.00 

4 





KSA x Course Packet 

0.065 

18.68 






KSA x Group 

0.009 

2.59 






Course Packet x Group 

0.002 

0.57 






Course Packet x Group 
x KSA 

0.1 19 

34.20 






D-study 






0.880 

0.873 


Table 32. Variance Components Estimation for Reading Group Judgments — Two-Facet Fully Crossed Design KSA Objective- 
Level Analysis 




Variance 






Generalizability Study 

Variance 

estimation 


Absolute 

Relative 



Variance component(s) 

estimation 

% 

Number 

error 

error 

P 


KSA Objective (KSA) 

0.201 

29.56 

37 

0.049 

0.039 

0.838 

0.804 

Course Packet 

0.022 

3.24 

8 





Group 

0.029 

4.26 

4 





KSA x Course Packet 

0.209 

30.74 






KSA x Group 

0.028 

4.12 






Course Packet x Group 

0.006 

0.88 






Course Packet x Group 
x KSA 

0.185 

27.21 






D-study 






0.808 

0.720 


than 90% of the estimates exceeded a higher 
threshold of .80; and more than 65% exceeded .85. 
The majority of variation came from the NAEP 
objectives and subtopics or standards, which was 
expected because each NAEP objective measures 
different KSAs. The main effect for content 
reviewers, either individual reviewers or groups of 
reviewers, accounted for less than 8% of the total 
variance in coding in all analyses. In many cases the 
main effect for reviewers, either individual reviewers 


or groups of reviewers, accounted for less than 5% of 
the total variance in coding. 

Likewise, course packets accounted for less than 7% 
of the total variance in all analyses. The Subtopic/ 
Standard x Course Packet interaction terms 
demonstrate that KSAs varied in their applicability 
across course packets. This is expected because four 
different course titles were represented in the eight 
validity course packets analyzed in both the 






Table 33. Variance Components Estimation for Mathematics Group Judgments — Two-Facet Fully Crossed Design Subtopic- 
Level Analysis 


Generalizability Study 
Variance component(s) 

Variance 

estimation 

Variance 

estimation 

% 

Number 

Absolute 

error 

Relative 

error 

P 

<t> 

Subtopic 

0.349 

56.38 

25 

0.025 

0.018 

0.952 

0.933 

Course Packet 

0.039 

6.30 

8 





Group 

0.010 

1.62 

4 





Subtopic x Course 

Packet 

0.273 

1 1.79 






Subtopic x Group 

0.018 

2.91 






Course Packet x Group 

0.006 

0.97 






Course Packet x Group 
x Subtopic 

0.124 

20.03 






D-study 






0.926 

0.897 


Table 34. Variance Components Estimation for Reading Group Judgments — Two-Facet Fully Crossed Design Standard-Level 
Analysis 




Variance 







Variance 

estimation 


Absolute 

Relative 



Variance component(s) 

estimation 

% 

Number 

error 

error 

P 


Standard 

0.294 

4.129 

10 

0.049 

0.037 

0.887 

0.856 

Course Packet 

0.006 

0.84 

8 





Group 

0.045 

6.32 

4 





Standard x Course 

Packet 

0.212 

29.78 






Standard x Group 

0.028 

3.93 






Course Packet x Group 

0.002 

0.28 






Course Packet x Group 
x Standard 

0.125 

17.56 






D-study 






0.868 

0.765 


mathematics and reading generalizability studies. 
Whether these judgments are being made by 
individual reviewers or by groups of reviewers, the 
data indicate the determinations are consistent, 
dependable, and reliable. 

Process Evaluations 

Process evaluation forms were administered after 
trainings in order to contribute information regarding 


the procedural validity of the study. The training 
process evaluation was offered to the entire pool of 
16 content reviewers in each subject area, including 
alternate reviewers. Overall, responses to training 
process evaluation questions were positive. More than 
90% of respondents indicated that they understood 
their role and felt comfortable with the training, and 
91% of mathematics respondents and 83% of reading 
respondents reported feeling well trained and 






prepared to complete the independent review. There 
were lower levels of agreement among the reading 
content reviewers (58%) that the independent review 
training webinar made the process clear; 64% of the 
mathematics respondents reported that time was used 
efficiently during the training. Respondents 
commented that they would have benefited from the 
“opportunity to work on training packets as a group” 
and it would have been “helpful to receive content 
feedback on the training packets.” 

Table 35 summarizes the percent of respondents 
providing “Agree” or “Strongly Agree” responses to 
each evaluation question posed at the conclusion of 
the content review qualifying course packet reviews 
and retraining. 

In the case of low levels of agreement that the 
independent review webinar training made the task clear 
to die content reviewers, the research team also 
provided individualized feedback to the reviewers as 
they mapped onto the packet review process after 


training packet reviews and the first qualifying packet 
review. This included email and phone conversations 
regarding any question the reviewers had after the 
training webinar and the training and qualifying reviews. 
The improved rates of alignment, as the reviewers 
completed the training and qualifying reviews, showed 
how the reviewers were able to map onto the coding 
process and exhibited increased understanding of the 
review task after the webinar. Additionally, the 
researchers addressed content reviewers’ questions or 
concerns at the group review meetings. 

At the conclusion of the independent review, the 
research team requested that the entire pool of 14 
content reviewers, including alternate content 
reviewers, complete an independent review process 
evaluation. Evaluations of the independent review 
process were positive with one exception. Both 
mathematics and reading respondents indicated that 
independent reviews of the course packets took 
longer than expected. Respondents’ comments 
showed varying opinions as to why it took longer. 


Table 35. Percentages of Content Reviewers Agreeing or Strongly Agreeing With Each Process Evaluation Statement 
Regarding Holistic Review and Independent Review Training 


Process evaluation statement 

Mathematics % 

Reading % 

The holistic review training webinar made the task clear to me. 

91 

75 

The holistic review training webinar was well organized. 

91 

75 

The independent review webinar training made the task clear to me. 

82 

58 

The independent review training webinars were well organized. 

82 

75 

The independent review training clearly identified the goals of the NAEP objective 
coding procedure. 

91 

83 

The independent review training clearly identified the goals of the NAEP objective 
annotation procedure. 

82 

100 

The training materials were helpful. 

91 

75 

1 am confident 1 understand my role. 

91 

92 

Time was used efficiently in the trainings. 

64 

75 

1 felt comfortable participating in the training webinars. 

91 

92 

Overall, 1 feel well trained and prepared to complete the independent review. 

91 

83 


Note. All content reviewers, including alternate content reviewers, were requested but not required to complete a training 
process evaluation. Eleven of fifteen mathematics reviewers and twelve of fourteen reading reviewers elected to do so. 




Two examples that capture the most common 
reasons are bulleted below: 

• Finding evidence was easy — knowing how to 
code them (and what to infer or not infer) was 
difficult and more time consuming than I 
expected. 

• The annotation process took longer than I 
thought it would. 

The findings from the process evaluation of the 
independent review process suggest that the 
reviewers’ level of comfort and confidence increased 
from the earlier stage of training. 


Table 36 summarizes the percentages of participants 
responding “Agree” or “Strongly Agree” to each 
evaluation item presented at the conclusion of the 
independent review. 

The group review evaluation was also positive. 
Almost all items had 100% agreement. In the two 
instances were there was not 100% agreement, 92% 
of reviewers agreed with the statement. Table 37 
summarizes the percentage of participants providing 
“Agree” or “Strongly Agree” responses to each of the 
evaluation items presented at the conclusion of the 
group review. 


Table 36. Percentages of Content Reviewers Agreeing or Strongly Agreeing With Each Process Evaluation Statement 
Regarding Independent Review 


Independent review process evaluation statement 

Mathematics % 

Reading % 

The training prepared me for independent review. 

93 

91 

The training materials were helpful. 

100 

100 

The holistic review process helped me code my packets during independent review. 

93 

82 

Referring to the NAEP frameworks helped me code my packets during independent 
review. 

93 

100 

Referring to the decision rules helped me code my packets during independent 
review. 

93 

100 

1 understood how to code the NAEP objectives in my course packets during 
independent review. 

100 

91 

Overall, 1 am satisfied with how 1 coded my packets during independent review. 

100 

100 

1 am satisfied with the support 1 received during the independent review. 

100 

100 

Overall, the independent review survey was easy to use. 

86 

91 

Overall, Basecamp was an effective communication tool for this project. 

100 

91 

1 found the to-dos in Basecamp useful for keeping track of packets during 
independent review. 

100 

100 

1 found it easy to access training materials, packets, and other files on Basecamp. 

100 

100 

During training and qualifying, 1 learned how to code the NAEP objectives as 1 
progressed, so my later training and qualifying coding may not be consistent with my 
earlier coding. 

71 

73 

The amount of work required to complete independent review was about what 1 
expected when 1 was recruited. 

50 

55 


Note. All content reviewers, including alternates, were requested but not required to complete an independent review process 
evaluation. Fourteen of fifteen mathematics reviewers and eleven of fourteen reading reviewers elected to do so. 




Table 37. Percentages of Content Reviewers Agreeing or Strongly Agreeing With Each Process Evaluation Statement 


Regarding Group Review 

Group review process evaluation question 

Mathematics % 

Reading % 

The orientation and training prepared me for group review. 

100 

92 

My notes and annotations were useful during group review. 

100 

100 

During group review, my group had enough time to discuss all of our packets. 

100 

100 

Overall, 1 think my group’s discussions were open and honest. 

100 

100 

Overall, 1 believe that my opinions were considered and valued by my group. 

100 

100 

1 feel this procedure was fair. 

100 

100 

My group followed the decision rules. 

100 

100 

1 am confident in the outcomes of the group review process. 

92 

100 

Overall, 1 am satisfied with my group’s final coding on our packets. 

100 

100 

The meeting was well organized. 

100 

100 


Note. All content reviewers who attended the group review were required to complete a group process evaluation. Twelve 
of twelve mathematics content reviewers and twelve of twelve reading content reviewers completed the evaluation. 


Overall, evaluation results suggest that training was 
effective and that content reviewers understood each 
task, were provided the necessary supports and 
materials for each task, felt the process was fair and 
collaborative, and were confident in study outcomes. 
These results provide support for the procedural 
validity of the study. 

Prerequisite Knowledge, Skills, and 
Abilities 

The purpose of the CCCA study was to identify the 
prerequisite KSAs for the work required in entry- 
level, credit-bearing college courses. As the reviewers 
considered the applicability of the NAEP objectives, 
they were permitted to disassemble the NAEP 
objectives to delineate between elements that were, 
and were not, evident in course materials. 
Understanding the KSA exclusions — terms or 
phrases in an objective that were not evident, and 
thus not prerequisite — is key to understanding and 
interpreting the elements of the NAEP frameworks 
that were prerequisite. 


Equally important to understanding entry-level 
college course prerequisites is the inclusion of non- 
NAEP additional KSAs that are evident in the course 
materials, but not in the NAEP frameworks. Thus the 
reviewers, based on the course artifacts, also 
recommended non-NAEP additional KSAs that were 
not captured within the NAEP objectives. 

Summaries of the KSA exclusions are presented prior 
to the prerequisite KSA descriptions. NAEP-specific 
KSAs and non-NAEP additional KSAs are presented 
for courses with substantial reading demands 
followed by the presentation of prerequisite KSAs for 
mathematics courses. 

Prerequisite NAEP-Specific KSA Exclusions 
Due to the structure of the NAEP frameworks, KSA 
exclusions could be made at either the subtopic/ 
standard or objective level in reading but only at the 
objective level in mathematics. All KSA exclusions 
identified in courses with substantial reading demands 
were made at the level of the standard: when the 
standard included the phrase “literary and 
informational text,” one of the two text descriptors 




was excluded depending on the course. For English 
literature, the only KSA exclusion identified was 
informational text and the only KSA exclusion for U.S. 
government, U.S. history, and psychology was literary 
text. These exclusions were not universal across every 
packet within a course title but represent a 
preponderance of the packets within a course title. 

The KSA exclusions in mathematics were generally 
one term or phrase of multipart NAEP objectives 
that contained prerequisites skills but also skills for 
which reviewers could find no evidence in course 
packets. The most common KSA exclusions across all 


course titles included logarithms, complex numbers, 
common irrational numbers, very large and very small 
numbers, and expressions and exponents. Table 38 
describes the mathematics KSA exclusions by course 
title. The table includes objectives that had a KSA 
exclusion recorded in at least one course tide. The 
KSA exclusion is the strikethrough portion of the 
objective. If the entire statement is listed without a 
strikethrough, then the objective was prerequisite to 
the course title without a KSA exclusion. If the 
objective was not prerequisite for the course title, this 
is noted in italics. 


Table 38. KSA Exclusions From NAEP Mathematics Objectives 


NAEP 

objective 

ID 

Precalculus/ 

calculus 

College algebra 

Finite mathematics 

Statistics 

l.l.d 

Represent, interpret, or 
compare expressions for 
real numbers, including 

nviM-rippirtnc i ipmrt 

Represent, interpret, or 
compare expressions for 
real numbers, including 
expressions using 
exponents and logarithms 

Represent, interpret, or 
compare expressions for 
real numbers, including 

AvriPApriAnp i ipin/r 

Represent, interpret, or 
compare expressions for 
real numbers, including 

A\/rM*nppip\np ■ ipmn 


LAUI L-. .3 .3 1 II J UjII 1^ 

exponents and logarithms 

LAUI L-. 3 .3 1 II j UjII lit 

exponents and logarithms 

LAUI LjjIwIIj UjII 1 if 

exponents and logarithms 

l.l.i 

Order or compare real 
numbers, including very 
large and very small real 

numbers 

Order or compare real 
numbers, including very 
large and very small real 

numbers 

Order or compare real 
numbers, including very 
large and very small real 

numbers 

Order or compare real 
numbers, including very 
large and very small real 

numbers 

l.3.b 

Perform arithmetic 
operations with real 
numbers, including 

/•rttYiivirtn i i~-i t- 1 /~, i-> -i 1 

Perform arithmetic 
operations with real 
numbers, including 

r/\rv\rY</\o irr-iti/innl 

Perform arithmetic 
operations with real 
numbers, including 

PAIYUVIAn i l~-l t- I /—, K-» A 1 

Perform arithmetic 
operations with real 
numbers, including 

rArYXYiAA i i- 1 -- 1 1 - 1 /—> r-\ -i 1 


LU 1 1 1 1 1 1 kj 1 1 ll I aliui lal 

numbers 

L.UI 1 II IIL/1 1 II 1 4LIUI lal 

numbers 

L.U 1 1 1 1 1 1 CM 1 1 1 1 ULILJI lal 

numbers 

L.UI 1 ll 1 iui 1 ll 1 duui idi 

numbers 

l.3.f 

Solve application 
problems involving 
numbers, including 
rational and common 
irrationals 

Solve application 
problems involving 
numbers, including 
rational and common 
irrationals 

Solve application 
problems involving 
numbers, including 
rational and common 
irrationals 

Solve application 
problems involving 
numbers, including 
rational and common 

irrationals 

l.4.c 

Use proportions to solve 
problems (including rates 
of change) 

Use proportions to solve 
problems (including rates 
of change) 

Use proportions to solve 
problems (including rates 
of change) 

Use proportions to solve 
problems (including rates 
of change) 

l.4.d 

Solve multistep problems 
involving percentages, 
including compound 
percentages 

Solve multistep problems 
involving percentages, 

in/“li mm/r r/MV»r>/^i in/4 

Solve multistep problems 
involving percentages, 
including compound 
percentages 

Solve multistep problems 
involving percentages, 
including compound 
percentages 


II IL.IUUII Ig IwVJI II UUU 1 1 u 

nni*rnnt--i /tap 


UL-l UU ILdrta 




Table 38. KSA Exclusions From NAEP Mathematics Objectives 


NAEP 

objective 

ID 

Precalculus/ 

calculus 

College algebra 

Finite mathematics 

Statistics 

l.5.c 

Solve problems using 
factors, multiples, or 
prime factorization 

Solve problems using 
factors, multiples, or 
prime factorization 

Solve problems using 
factors, multiples, or 
prime factorization 

Solve problems using 
factors, multiples, or 
prime factorization 





Objective not 
prerequisite for course 
title 

l.5.d 

Use divisibility or 
remainders in problem 
settings 

Use divisibility or 
remainders in problem 
settings 

Use divisibility or 
remainders in problem 
settings 

l.5.f 

Recognize properties of 
the number system 
(whole numbers, integers, 
rational numbers, real 
numbers, and complex 
numbers) and how they 
arc related to each other, 

and identify examples of 
each type of number 

Recognize properties of 
the number system 
(whole numbers, integers, 
rational numbers, real 
numbers, and complex 
numbers) and how they 
arc related to each other, 

and identify examples of 
each type of number 

Recognize properties of 
the number system 
(whole numbers, integers, 
rational numbers, real 
numbers, and complex 
numbers) and how they 
arc related to each other, 

and identify examples of 
each type of number 

Objective not 
prerequisite for course 
title 

2.2.d 

Objective not 
prerequisite for course 
title 

Objective not 
prerequisite for course 
title 

Understand that 
numerical values 
associated with 
measurements of physical 
quantities are 

Objective not 
prerequisite for course 
title 


to variation, and must be 
assigned units of 
measurement 

2.3.e 

*Determine the radian 
measure of an angle ae4 
explain how radian 
measurement is related 

to a circle of radius 1 

Objective not 
prerequisite for course 
title 

Objective not 
prerequisite for course 
title 

Objective not 
prerequisite for course 
title 

3. 2. a 

Recognize or identify 
types of symmetries (e.g., 
point, line, rotational, 
self-congruence) of two- 

Objective not 
prerequisite for course 
title 

Objective not 
prerequisite for course 
title 

Objective not 
prerequisite for course 
title 


figures 




3.3.b 

Apply geometric 
properties and 
relationships to solve 
problems in two and 
three dimensions 

Apply geometric 
properties and 
relationships to solve 
problems in two-amd 
three dimensions 

Objective not 
prerequisite for course 
title 

Objective not 
prerequisite for course 
title 

3.3.d 

Use the Pythagorean 
theorem to solve 
problems in two- or 
three-dimensional 
situations 

Use the Pythagorean 
theorem to solve 
problems in two- or 
three-dimensional 
situations 

Use the Pythagorean 
theorem to solve 
problems in two- or 
three-dimensional 
situations 

Objective not 
prerequisite for course 
title 




Table 38. KSA Exclusions From NAEP Mathematics Objectives 


NAEP 

objective 

ID 

Precalculus/ 

calculus 

College algebra 

Finite mathematics 

Statistics 

3.3.e 

Recall and interpret 
definitions and basic 
properties of congruent 
and similar triangles, 

circles, quadrilaterals, 
polygons, parallel, 
perpendicular arrd 
intersecting lines, and 

associated angle 

relationships 

Recall and interpret 
definitions and basic 
properties of congruent 
and similar triangles, 
circles, quadrilaterals, 
polygons, parallel, 

perpendicular and 
intersecting lines, and 

associated angle 

relationships 

Objective not 
prerequisite for course 
title 

Objective not 
prerequisite for course 
title 

3.3.g 

Analyze properties and 
relationships of parallel, 
perpendicular, or 
intersecting lines including 
the angle relationships 
that arise in these cases 

Objective not 
prerequisite for course 
title 

Analyze properties and 
relationships of parallel, 
perpendicular, or 
intersecting lines including 
the angle relationships 
that arise in these cases 

Objective not 
prerequisite for course 
title 

3.4.g 

*Graph ellipses an4 
hyperbolas whose axes 
are parallel to the 
coordinate axes and 
demonstrate 
understanding of the 
relationship between 
their standard algebraic 
form and their graphical 
characteristics 

Objective not 
prerequisite for course 
title 

Objective not 
prerequisite for course 
title 

Objective not 
prerequisite for course 
title 

4. 1.b 

For a given set of data, 
complete a graph and 
solve a problem using the 
data in the graph 
(histograms, scatterplots, 
and line graphs) 

Objective not 
prerequisite for course 
title 

For a given set of data, 
complete a graph and 
solve a problem using the 
data in the graph 
(histograms, scatterplots, 
and line graphs) 

For a given set of data, 
complete a graph and 
solve a problem using the 
data in the graph 
(histograms, scatterplots, 
and line graphs) 


Express linear and 
exponential functions in 

Objective not 

Express linear and 
exponential functions in 
recursive and explicit 
form given a table, verbal 
description, or some 
terms of a sequence 

Objective not 

5. 1.b 

form given a table, verbal 
description, or some 
terms of a sequence 

prerequisite for course 
title 

prerequisite for course 
title 

5.1.1 

Determine the domain 
and range of functions 
given in various forms and 
contexts 

Determine the domain 
and range of functions 
given in various forms and 
contexts 

Determine the domain 
and range of functions 
given in various forms and 
contexts 

Objective not 
prerequisite for course 
title 

5-l-j 

*Given a function, 
determine its inverse if it 
exists and explain the 

/■/-int-AN/fi i^l tviA'inirwr /—> + Li /-\ 

Objective not 
prerequisite for course 

Objective not 
prerequisite for course 

Objective not 
prerequisite for course 


UJIILLAlUUI 1 1 IL.d.1 III li: Ul UIL 

inverse for a given 

situation 

title 

title 

title 




Table 38. KSA Exclusions From NAEP Mathematics Objectives 


NAEP 

objective 

ID 

Precalculus/ 

calculus 

College algebra 

Finite mathematics 

Statistics 

5. 2. a 

Create and translate 
between different 
representations of 
algebraic expressions, 
equations, and inequalities 
(e.g., linear, quadratic, 
exponential, or 
trigonometric) using 
symbols, graphs, tables, 
diagrams, or written 
descriptions 

Create and translate 
between different 
representations of 
algebraic expressions, 
equations, and inequalities 
(e.g., linear, quadratic, 
exponential, or 
trigonometric) using 
symbols, graphs, tables, 
diagrams, or written 
descriptions 

Create and translate 
between different 
representations of 
algebraic expressions, 
equations, and inequalities 
(e.g., linear, quadratic, 
exponential, or 
trigonometric) using 
symbols, graphs, tables, 
diagrams, or written 
descriptions 

Objective not 
prerequisite for course 
title 

5.2. b 

Analyze or interpret 
relationships expressed in 
symbols, graphs, tables, 
diagrams (including Venn 
diagrams), or written 
descriptions and evaluate 
the relative advantages or 

disadvantages of different 

representations to 
answer specific questions 

Analyze or interpret 
relationships expressed in 
symbols, graphs, tables, 
diagrams (including Venn 
diagrams), or written 
descriptions and evaluate 
the relative advantages or 
disadvantages of different 
representations to 
answer specific questions 

Analyze or interpret 
relationships expressed in 
symbols, graphs, tables, 
diagrams (including Venn 
diagrams), or written 
descriptions and evaluate 
the relative advantages or 

disadvantages of different 

representations to 
answer specific questions 

Objective not 
prerequisite for course 
title 

5.2.d 

Perform or interpret 
transformations on the 
graphs of linear, 
quadratic, exponential, 
and trigonometric 
functions 

Perform or interpret 
transformations on the 
graphs of linear, 
quadratic, exponential, 
and trigonometric 

functions 

Objective not 
prerequisite for course 
title 

Objective not 
prerequisite for course 
title 

5.3.b 

Write algebraic 
expressions, equations, or 
inequalities to represent a 
situation 

Write algebraic 
expressions, equations, or 
inequalities to represent a 
situation 

Write algebraic 
expressions, equations, or 
inequalities to represent a 
situation 

Objective not 
prerequisite for course 
title 

5.3.c 

Perform basic operations, 
using appropriate tools, 
on algebraic expressions 
including polynomial an4 

Perform basic operations, 
using appropriate tools, 
on algebraic expressions 
including polynomial an4 

Perform basic operations, 
using appropriate tools, 
on algebraic expressions 
including polynomial and 

Perform basic operations, 
using appropriate tools, 
on algebraic expressions 
including polynomial and 
rational expressions 


[ dUUI Icll LAU 1 CoolOllo 

1 uLIUI Icll LAUI CoolC/llo 

1 clLICJI Icll LAUI jlvl 1 j 

5.3.d 

Write equivalent forms of 
algebraic expressions, 
equations, or inequalities 
to represent and explain 
mathematical 
relationships 

Write equivalent forms of 
algebraic expressions, 
equations, or inequalities 
to represent and explain 
mathematical 
relationships 

Write equivalent forms of 
algebraic expressions, 
equations, or inequalities 
to represent and explain 
mathematical 
relationships 

Objective not 
prerequisite for course 
title 

5.3.e 

Evaluate algebraic 
expressions including 
polynomials and rational 
expressions 

Evaluate algebraic 
expressions including 
polynomials and rational 
expressions 

Evaluate algebraic 
expressions including 
polynomials and rational 

n\/rM*Arpiftr\r 

Evaluate algebraic 
expressions including 
polynomials and rational 
expressions 


LAUI C. 1 LJ 1 1 




Table 38. KSA Exclusions From NAEP Mathematics Objectives 


NAEP 

objective 

ID 

Precalculus/ 

calculus 

College algebra 

Finite mathematics 

Statistics 

5.3.f 

Use function notation to 
evaluate a function at a 
specified point in its 
domain and combine 
functions by addition, 
subtraction, 
multiplication, division, 
and composition 

Use function notation to 
evaluate a function at a 
specified point in its 
domain and combine 
functions by addition, 
subtraction, 
multiplication, division, 
and composition 

Use function notation to 
evaluate a function at a 
specified point in its 
domain and combine 
functions by addition, 
subtraction, 

Use function notation to 
evaluate a function at a 
specified point in its 
domain and combine 
functions by addition, 

subtraction, 


and composition 

and composition 

5.3.h 

Use basic properties of 
exponents and 
’'logarithms to solve 
problems 

Use basic properties of 
exponents and 
’'logarithms to solve 
problems 

Objective not 
prerequisite for course 
title 

Objective not 
prerequisite for course 
title 

5. 4. a 

Solve linear, rational, or 
quadratic equations or 
inequalities, including 
those involving absolute 

value 

Solve linear, rational, or 
quadratic equations or 
inequalities, including 
those involving absolute 

value 

Solve linear, rational, or 
quadratic equations or 
inequalities, including 
those involving absolute 

value 

Solve linear, rational, or 
quadratic equations or 
inequalities, including 
those involving absolute 

value 


Note. The exclusions in this table represent varying frequencies across packets. For more information, see the mathematics 
content maps in Appendix R. An asterisk (*) denotes objectives that describe mathematics content beyond that typically 
taught in a standard 3-year course of study (the equivalent of I year of geometry and 2 years of algebra). Therefore, these 
objectives will be selected less often than the others for inclusion on the NAEP assessments. 


Non-NAEP Additional KSAs 

One goal of the CCCA study was to identify 
prerequisite KSAs not found in the current NAEP 
mathematics and reading frameworks. The narrative 
below describes the final list of non-NAEP additional 
KSAs identified for consideration by the NAEP experts. 

For group reviews, the research team compiled by 
course packet the list of non-NAEP additional KSAs 
generated during independent review. Included in the 
list were all the non-NAEP additional KSAs for that 
packet identified by each reviewer. EPIC cleaned each 
list to remove duplicates noted by the following: 

• Non-NAEP additional KSAs with identical 
phrasing were collapsed into a single KSA. 

• Almost-identical duplication of phrasing such 
as “Discern credibility of source” vs. “Discern 
credibility of a source” was combined into a 
single KSA. 


The researchers were conservative in the elimination 
of duplicates; if there was any doubt regarding 
whether two statements were duplicates, the two 
statements were left in the list for the groups to 
review. Eighty-two non-NAEP additional KSAs were 
generated during independent review for review by 
the mathematics content reviewer groups. Seventy-six 
non-NAEP additional KSAs were generated during 
independent review for review by the reading content 
reviewer groups. 

Each non-NAEP additional KSA was reviewed 
within the context of each of the 28 packets. The 
following are explanations for how non-NAEP 
additional KSAs were eliminated during group review. 

• Non-NAEP additional KSAs were not direcdy 
related to mathematics or reading KSAs. 

• Groups determined the KSA could be found 
within the NAEP framework. 




• In the group reviews, content reviewers 
discussed and shared their expertise and 
annotated evidence. In this process, content 
reviewers could change their mind on a non- 
NAEP additional KSA (see the Reasons for 
Changes in Coding Decisions section). 

• Groups had the opportunity to edit the non- 
NAEP additional KSA to better reflect the 
intended meaning. Facilitators recorded the 
edits. All were discussed thoroughly in the 
group reviews and all 28 packets were coded 
using any new non-NAEP additional KSA. 
Below are examples of the edited non-NAEP 
additional KSAs recommended in both 
reading and mathematics. 

o One mathematics group indicated that 
“Use the Cartesian coordinate system” 
and “Basic properties of the coordinate 
plane including graphing sets of points” 
were the same skill and could be 
combined. 

o One reading group edited “Distinguish 
between primary and secondary sources” 
to “Locate and distinguish between 
primary and secondary sources.” 

The group review culminated in a list of 16 non-NAEP 
additional KSAs across the courses with substantial 
reading demands and 1 5 non-NAEP additional KSAs 
across the mathematics courses. These lists were 
brought forward to the NAEP expert reviews for 
consideration. These non-NAEP additional KSAs are 
presented in the following sections after the KSAs that 
are related to NAEP objectives. 

Prerequisite KSAs for Courses With Substantial 
Reading Demands 

Table 39 summarizes, by KSA, the percentages of 
course packets for which the KSA was considered to 
be a prerequisite, within and across the four courses 
with substantial reading requirements. A KSA was 


considered to be a prerequisite if, for at least one 
course packet, it was identified as prerequisite during 
independent review and agreed upon as a prerequisite 
at the group review. Highly qualified and trained 
content reviewers identified the prerequisite KSAs. 

Prerequisite KSAs from the NAEP reading 
framework. The KSAs that appear to be most 
ubiquitous in their applicability as prerequisite to 
college course entry include the following five, which 
are considered to be foundational for reading 
comprehension at the college level: 

1 . Locate and recall specific information such as 
definitions, facts, and supporting details 

(LI. a.) 

2. Describe a problem and solution or cause and 
effect (2. l.a.) 

3. Determine word meaning (2.4.a.) 

4. Take different perspectives in relation to a 
text (3.1.c.) 

5. Compare or connect ideas, perspectives, 
problems, or situations (2. Lb.) 

The KSAs found to be prerequisite generally differed 
between course titles that relied heavily on literary texts 
(English literature) and course titles that relied heavily 
on informational texts (psychology, U.S. history, U.S. 
government). The KSAs relating to locating or 
recalling textually explicit information and making 
simple inferences within and across literary texts 
(standard 1 .2) were considered prerequisite in most 
English literature course packets but rarely in course 
packets within other course titles. Also prerequisite to 
almost all of the English literature course packets (and 
very few packets from other courses) was the ability to 
integrate ideas to determine theme (2.2.b.). This is 
expected, given the group reviewers’ determination 
that literary texts, and therefore KSAs related to literary 
texts, were more relevant to English literature than to 
the other course titles. 


Table 39. Percentages of Course Packets in Which the Reading Objective Was Considered a Prerequisite Across all Courses 
and by Course Title 


Objective 

Overall 

% 

English 

literature 

% 

Psychology 

% 

u.s. 

government 

% 

U.S. 

history 

% 

1. Locate/recall: Locate or recall textually explicit information within and across texts, which may involve making 
simple inferences as needed for literal comprehension 

l .l. Locate or recall textually explicit information and make 
informational texts 

simple inferences within and across 

both literary and 


l.l. a. Locate or recall specific 
information such as definitions, facts, 
and supporting details in text or 
graphics 

100 

100 

100 

100 

100 

1.2. Locate or recall textually explicit information and make 

simple inferences within and across 

literary texts 


1 .2. a. Locate or recall character traits 

24 

85 

0 

0 

10 

l.2.b. Locate or recall sequence of 
events or actions 

26 

90 

0 

0 

15 

l.2.c. Locate or recall setting 

25 

85 

0 

0 

15 

l.2.d. Locate or recall figurative language 

20 

80 

0 

0 

0 

l.2.e. Locate or recall organizing 
structures of literary texts, such as 
verse or stanza in poetry or description, 
chronology, comparison, etcetera in 
literary nonfiction 

21 

80 

0 

0 

5 

1.3. Locate or recall textually explicit information and make 

simple inferences within and across 

informational texts 


1.3. a. Locate or recall the topic 
sentence or main idea 

88 

55 

100 

95 

100 

1 .3.b. Locate or recall the author's 
purpose 

69 

35 

80 

85 

75 

l.3.c. Locate or recall causal relations 

86 

47 

100 

100 

95 

l.3.d. Locate or recall organizing 
structures of texts, such as 
comparison/contrast, problem/solution, 
enumeration, etc. 

76 

30 

95 

85 

95 

2. Integrate/interpret: Make complex inferences within and across texts 




2.1. Make complex inferences within and across 

both literary and informational texts 



2.1. a. Describe problem and solution or 
cause and effect 

91 

85 

95 

95 

90 

2. 1 .b. Compare or connect ideas, 
perspectives, problems, or situations 

99 

100 

100 

95 

100 

2. 1.c. Determine unstated assumptions 
in an argument 

69 

55 

65 

75 

80 

2. 1 .d. Describe or analyze how an 
author uses literary devices or text 

64 

80 

70 

55 

50 


features to convey meaning 




Table 39. Percentages of Course Packets in Which the Reading Objective Was Considered a Prerequisite Across all Courses 


and by Course Title 


Objective 

Overall 

% 

English 

literature 

% 

Psychology 

% 

u.s. 

government 

% 

U.S. 

history 

% 

2. 1 .e. Describe or analyze how an 
author uses organizing structures to 
convey meaning 

59 

85 

40 

50 

60 

2. 1 .f. Describe or analyze author’s 
purpose 

78 

90 

80 

65 

75 

2.2. Make complex inferences within and across 

literary texts 





2. 2. a. Interpret mood, tone, or voice 

19 

70 

0 

0 

5 

2.2.b. Integrate ideas to determine 
theme 

26 

95 

0 

0 

10 

2.2.c. Interpret a character's conflicts, 
motivations, and decisions 

16 

55 

0 

0 

10 

2.2.d. Examine relations between or 
among theme, setting, plot, or 
characters 

19 

65 

0 

0 

10 

2.2.e. Explain how rhythm, rhyme, 
sound, or form in poetry contribute to 
meaning 

14 

55 

0 

0 

0 

2.3. Make complex inferences within and across 

informational texts 




2. 3. a. Summarize major ideas 

84 

45 

100 

90 

100 

2.3.b. Draw conclusions and provide 
supporting information 

76 

35 

95 

85 

90 

2.3.c. Find evidence in support of an 
argument 

78 

35 

100 

85 

95 

2.3.d. Distinguish facts from opinions 

68 

15 

80 

85 

90 

2.3.e. Determine the importance of 
information within and across texts 

81 

35 

100 

95 

95 

2.4. Apply understanding of vocabulary to comprehension of both literary and 

informational texts 


2.4. a. Determine word meaning as used 
in context 

88 

80 

100 

75 

95 

3. Critique/evaluate: Consider text(s) critically 

3.1. Consider both literary and informational texts critically 





3. 1 .a. Judge the author’s craft and 
technique 

33 

65 

15 

15 

35 

3. 1 .b. Analyze, critique, or evaluate the 
author’s perspective or point of view 

68 

60 

60 

75 

75 

3.l.c. Take different perspectives in 
relation to a text 

84 

75 

80 

90 

90 




Table 39. Percentages of Course Packets in Which the Reading Objective Was Considered a Prerequisite Across all Courses 
and by Course Title 


Objective 

Overall 

% 

English 

literature 

% 

Psychology 

% 

U.S. 

government 

% 

U.S. 

history 

% 

3.2. Consider literary texts critically 






3. 2. a. Evaluate the role of literary 
devices in conveying meaning 

16 

65 

0 

0 

0 

3.2.b. Determine the degree to which 
literary devices enhance a literary work 

10 

40 

0 

0 

0 

3.2.c. Evaluate a character’s conflict, 
motivations, and decisions 

15 

50 

0 

0 

10 

3.3. Consider informational texts critically 






3. 3. a. Evaluate the way the author 
selects language to influence readers 

23 

15 

10 

40 

25 

3.3.b. Evaluate the strength and quality 
of evidence used by the author to 
support his or her position 

58 

15 

70 

75 

70 

3.3.c. Determine the quality of 
counterarguments within and across 
texts 

63 

20 

65 

85 

80 

3.3.d. Judge the coherence or logic of an 
argument 

65 

20 

80 

85 

75 


The sampled course titles that rely heavily on 
informational texts (i.e., psychology, U.S. government, 
U.S. history) had similar prerequisite KSAs, with U.S. 
history demonstrating slight differences in KSAs 
related to literary texts across the standards. The ability 
to locate or recall textually explicit information, 
including topic sentence, author’s purpose, causal 
relations, and organizing structures, and make simple 
inferences within and across informational texts (1.3. a. 
- 1.3.d.) were considered prerequisite in a majority of 
course packets from courses that rely heavily on 
informational texts. Specific abilities (2.3.a - 2.3.e) 
related to making complex inferences within and 
across informational texts were coded as prerequisite in 
almost all of these course packets. The ability to judge 
the coherence or logic of an argument (3.3. d.) was also 
commonly considered a prerequisite for the course 
titles that rely heavily on informational texts. 


KSAs related to the ability to critique or evaluate 
literary and informational texts (standard 3.1) were 
considered prerequisite within the course packets 
across course titles. The ability to consider literary 
texts literally (standard 3.2) was not considered to be 
prerequisite for psychology or U.S. government; it 
was considered prerequisite in a minority of U.S. 
history course packets. Other KSAs that were 
considered prerequisite in some English literature 
course packets but not others (and rarely considered 
prerequisite in psychology, U.S. government, or U.S. 
history packets) included most KSAs relating to 
making complex inferences within and across literary 
texts (standard 2.2), and considering literary texts 
critically (standard 3.2). Taking different perspectives 
in relation to literature (3.1.c.) was found to be a 
frequent prerequisite for English literature course 
packets and across all other course titles. 




Overall, fewer course packets across the English 
literature, U.S. history, U.S. government, and 
psychology course titles were found to contain 
prerequisite KSAs from within the following standards: 

• Locate or recall textually explicit information 
and make simple inferences within and across 
literary texts (standard 1 .2) 

• Make complex inferences within and across 
literary texts (standard 2.2) 

• Consider literary texts critically (standard 3.2) 

Table 39 is color-coded to show KSAs considered to 
be prerequisite in 25% or fewer of the course packets 
(shown in orange), KSAs considered prerequisite in 
26% to 74% of packets (shown in blue), and KSAs 
considered prerequisite in 75% or more of the 
packets (shown in green). 

The group review generated 1 6 prerequisite KSAs 
that were not included in the NAEP framework. 
These non-NAEP additional KSAs are presented in 
Table 40 in terms of the frequency with which 
content reviewers, during group review, identified 
them in course packets overall and by course. The 
most commonly identified KSAs that were not 
represented within the NAEP framework related to a 
student’s ability to interact with text and resources in 
an online environment, apply course material to world 
events and contexts, critique the significance of the 
context and time period in which an author wrote, 
discern source credibility and process research 
materials, and analyze both factual and theoretical 
information in informational text. All commonly 
identified non-NAEP additional KSAs were found in 
course packets across course titles. 

Prerequisite KSAs for Mathematics Courses 
Given the number of mathematics objectives in the 
NAEP framework, the presentation of prerequisite 
KSAs for mathematics courses is organized into 


separate data tables by subject area, followed by a 
presentation of the non-NAEP additional KSAs that 
are not represented in the NAEP framework. 

Across all mathematics course titles, the Number 
Properties and Operations subject area of the NAEP 
framework was the area from which the most 
prerequisites were identified. The ability to represent, 
interpret, or compare expressions for real numbers, 
including exponents, was the most ubiquitous 
prerequisite KSA for college-level coursework. The 
ability to perform arithmetic operations with real 
numbers, including common irrational numbers 
(1.3.b.), and apply basic properties of operations, 
including conventions about the order of operations 
(1.5.e.), were considered prerequisite KSAs for 
significant proportions of course packets within all 
sampled course titles except statistics. KSAs related to 
ratios and proportional reasoning were considered 
prerequisite only for some (26%) precalculus /calculus 
courses. KSAs relating to estimation or mathematical 
reasoning subtopics in the NAEP framework were 
considered prerequisite in only a small proportion of 
course packets. 

Many NAEP objectives in die subject area of Number 
Properties and Operations were considered prerequisite in 
some courses and not in others. Whether the following 
KSAs were considered prerequisite varied by course 
packet: ability to order or compare real numbers; 
describe the effect of multiplying and dividing by 
numbers (including by 0, less than 0, between 0 and 1 , 
and greater than 1); solve application problems 
involving numbers (including rational and common 
irrationals); solve problems using factors, multiples, or 
prime factorization; apply basic properties of operations; 
and recognize properties of the number system and how 
they relate to each other. These KSAs were, however, 
considered prerequisite in more precalculus/ calculus 
and college algebra course packets than in finite 
mathematics or statistics course packets. 


Table 40. Non-NAEP Additional KSAs for Courses With Substantial Reading Demands 


Non-NAEP additional prerequisite KSA 

Number of 

course 

packets 

English 

literature 

Psychology 

u.s. 

government 

U.S. 

history 

Apply course material to think about the world 

19 

5 

4 

5 

5 

Critique/evaluate the significance of the historical 
or cultural context of a text or author 

18 

5 

4 

4 

5 

Locate or recall textually explicit information 
through a course website or other websites 

14 

2 

4 

4 

4 

Have some knowledge of interacting with 
reading/text in an online environment 

14 

3 

4 

4 

3 

Discern credibility of source 

12 

1 

5 

4 

2 

Read and process research materials 

12 

2 

2 

4 

4 

Analyze both factual and theoretical information 
in informational text 

1 1 

2 

5 

4 

0 

Utilize text features (e.g., glossaries, key terms, 
key ideas, or issues that are highlighted) or 
online links (e.g., online study center, charts, or 
technology sections) as an aid to reading 

1 1 

0 

5 

3 

3 

Integrate text with personal experience 

7 

1 

4 

2 

0 

Locate and distinguish between primary and 
secondary sources 

7 

1 

3 

1 

2 

Understand various theoretical models of 
domain-specific content (e.g., historical, feminist 
response, or ecocriticism) 

6 

0 

5 

1 

0 

Distinguish between primary text sources and 
secondary text sources 

4 

0 

0 

0 

4 

Analyze complex literary or informational text 

3 

0 

0 

0 

3 

Recognize complexities of the underlying 
traditions in a literary text (e.g., Romanticism, 
Modernism, or Transcendentalism) 

2 

2 

0 

0 

0 

Able to read English well 

2 

0 

1 

1 

0 

Able to read texts aloud in class to engage in the 
class discussion and interpretation 

1 

1 

0 

0 

0 


The following tables, representing each mathematics 
content area, are color-coded to show KSAs that were 
considered to be prerequisite in 25% or less of the 
course packets (shown in orange), KSAs that were 


considered prerequisite in 26% to 74% of packets 
(shown in blue), and KSAs that were considered 
prerequisite in 7 5% or more of the packets (shown in 
green). See Table 41. 




Table 4 1 . Percentages of Course Packets in Which the Mathematics Objectives Under Number Properties and Operations 
Were Considered a Prerequisite Across all Courses and by Course Title 


Objective 

Overall 

% 

Precalculus/ 

calculus 

% 

College 

algebra 

% 

Finite 

mathematics 

% 

Statistics 

% 

1. Number properties and operations 






1 . 1 . Number sense 






1 . 1 .d. Represent, interpret, or 
compare expressions for real numbers, 
including expressions using exponents 
and logarithms 

92 

100 

90 

95 

85 

1 . 1 .f. Represent or interpret 
expressions involving very large or 
very small numbers in scientific 
notation 

20 

0 

65 

10 

5 

l.l.g. Represent, interpret, or compare 
expressions or problem situations 
involving absolute values 

34 

63 

75 

0 

0 

1 . 1 .i. Order or compare real numbers, 
including very large and very small real 
numbers 

73 

79 

95 

60 

60 

1.2. Estimation 






1 .2.b. Identify situations where 
estimation is appropriate, determine 
the needed degree of accuracy, and 
analyze* the effect of the estimation 
method on the accuracy of results 

1 

0 

0 

5 

0 

1 .2.c. Verify solutions or determine the 
reasonableness of results in a variety of 
situations 

10 

21 

15 

5 

0 

1 .2.d. Estimate square or cube roots of 
numbers less than 1 ,000 between two 
whole numbers 

0 

0 

0 

0 

0 

1.3. Number operations 






1.3. a. Find integral or simple fractional 
powers of real numbers 

62 

95 

90 

40 

0 

l.3.b. Perform arithmetic operations 
with real numbers, including common 
irrational numbers 

97 

100 

100 

90 

0 

l.3.c. Perform arithmetic operations 
with expressions involving absolute 
value 

34 

63 

75 

0 

0 

1 .3.d. Describe the effect of multiplying 
and dividing by numbers including the 
effect of multiplying or dividing a real 
number by 0, or a number less than 0, 

19 

26 

37 

10 

0 


or a number between 0 and I , I , or a 
number greater than I 




Table 41. Percentages of Course Packets in Which the Mathematics Objectives Under Number Properties and Operations 
Were Considered a Prerequisite Across all Courses and by Course Title 


Objective 

Overall 

% 

Precalculus/ 

calculus 

% 

College 

algebra 

% 

Finite 

mathematics 

% 

Statistics 

% 

l.3.f. Solve application problems 
involving numbers, including rational 
and common irrationals 

63 

89 

85 

40 

30 

1 .4. Ratios and proportional reasoning 






1 .4.c. Use proportions to solve 
problems (including rates of change) 

24 

26 

20 

20 

25 

l.4.d. Solve multistep problems 
involving percentages, including 
compound percentages 

19 

26 

25 

15 

0 

1.5. Properties of number and operations 






l.5.c. Solve problems using factors, 
multiples, or prime factorization 

47 

89 

85 

10 

0 

l.5.d. Use divisibility or remainders in 
problem settings 

8 

16 

10 

5 

0 

l.5.e. Apply basic properties of 
operations, including conventions 
about the order of operations 

89 

95 

95 

85 

0 

1 .5.f. Recognize properties of the 
number system (whole numbers, 
integers, rational numbers, real 
numbers, and complex numbers) and 
how they are related to each other 
and identify examples of each type of 
number 

28 

42 

50 

20 

0 

1.6. Mathematical reasoning using number 






1.6. a. *Give a mathematical argument 
to establish the validity of a simple 
numerical property or relationship 

3 

1 1 

0 

0 

0 

1 .6.b. ^Analyze or interpret a proof by 
mathematical induction of a simple 
numerical relationship 

0 

0 

0 

0 

0 


Note. An asterisk (*) denotes objectives that describe mathematics content beyond that typically taught in a standard 3-year 
course of study (the equivalent of I year of geometry and 2 years of algebra). Therefore, these objectives will be selected less 
often than the others for inclusion on the NAEP assessments. 


All of the KSAs considered prerequisite under the 
Measurement subject area of the NAEP Mathematics 
framework were found in precalculus/ calculus, 
college algebra, and finite mathematics course 
packets. There were six KSAs represented by 
objectives that were considered to be prerequisite in 


between a fourth and a third of the 
precalculus/calculus course packets (2.1. f, 2.1.h, 2.3. c, 
2.3.d, 2.3. £). The majority of these related to 
trigonometry. A few other Measurement KSAs were 
found in a small minority of precalculus/ calculus 
course packets; for instance, solve problems involving 




rates such as speed, density, population density, or 
flow rates (2.1.i.), was found to be an applicable 
prerequisite in 21% of precalculus/ calculus packets. 
Some Measurement prerequisites were found in college 
algebra and finite mathematics course packets, 
although less frequently than were found in the 
precalculus/calculus course packets. The most 
commonly determined prerequisites for college 
algebra involved solving problems involving 


perimeter or area of plane figures (2.1.E); solving 
problems by determining, estimating, or comparing 
volumes or surface areas of three-dimensional figures 
(2.1.h.); and solving problems involving indirect 
measurement (2.3. a.). Three KSAs were determined 
to be prerequisite in one finite mathematics packet 
each (2.1.h, 2.2.b, 2.2.d). There were no Measurement 
KSAs considered prerequisite in the sampled statistics 
course packets. See Table 42. 


Table 42. Percentages of Course Packets in Which the Mathematics Objectives Under Measurement Were Considered a 
Prerequisite Across all Courses and by Course Title 




Precalculus/ 

College 

Finite 



Overall 

calculus 

algebra 

mathematics 

Statistics 

Objective 

% 

% 

% 

% 

% 

2. Measurement 






2.1. Measuring physical attributes 






2. 1 .b. Determine the effect of 






proportions and scaling on length, area, 
and volume 

0 

0 

0 

0 

0 

2. 1.c. Estimate or compare perimeters 
or areas of two-dimensional geometric 
figures 

1 

0 

5 

0 

0 

2. 1.d. Solve problems of angle measure, 
including those involving triangles or 
other polygons or parallel lines cut by a 
transversal 

1 

5 

0 

0 

0 

2. 1.f. Solve problems involving 
perimeter or area of plane figures such 
as polygons, circles, or composite 
figures 

16 

32 

35 

0 

0 

2. 1.h. Solve problems by determining, 
estimating, or comparing volumes or 
surface areas of three-dimensional 
figures 

14 

26 

25 

5 

0 

2. 1.i. Solve problems involving rates 
such as speed, density, population 
density, or flow rates 

10 

21 

20 

0 

0 

2.2. Systems of measurement 






2. 2. a. Recognize that geometric 
measurements (length, area, perimeter, 
and volume) depend on the choice of a 

0 

0 

0 

0 

0 


unit and apply such units in expressions, 
equations, and problem solutions 




Table 42. Percentages of Course Packets in Which the Mathematics Objectives Under Measurement Were Considered a 
Prerequisite Across all Courses and by Course Title 


Objective 

Overall 

% 

Precalculus/ 

calculus 

% 

College 

algebra 

% 

Finite 

mathematics 

% 

Statistics 

% 

2.2. b. Solve problems involving 
conversions within or between 
measurement systems, given the 
relationship between the units 

2.2. d. Understand that numerical values 
associated with measurements of physical 

1 

0 

0 

5 

0 

quantities are approximate, are subject 
to variation, and must be assigned units 
of measurement 

2.2.e. Determine appropriate accuracy of 
measurement in problem situations (e.g., 

1 

0 

0 

5 

0 

the accuracy of measurement of the 
dimensions to obtain a specified accuracy 
of area) and find the measure to that 
degree of accuracy 

0 

0 

0 

0 

0 

2.2.f. Construct or solve problems 
involving scale drawings 

0 

0 

0 

0 

0 

2.3. Measurement in triangles 

2.3.a. Solve problems involving indirect 
measurement 

9 

1 1 

25 

0 

0 

2.3.b. Solve problems using the fact that 

trigonometric ratios (sine, cosine, and 
tangent) stay constant in similar triangles 

4 

16 

0 

0 

0 

2.3.c. Use the definitions of sine, cosine, 

and tangent as ratios of sides in a right 
triangle to solve problems about length 
of sides and measure of angles 

2.3.d. Interpret and use the identity sin 2 q 
+ cos 2 q = 1 for angles q between 0° and 

8 

32 

0 

0 

0 

90°; recognize this identity as a special 
representation of the Pythagorean 
theorem 

6 

26 

0 

0 

0 

2.3.e. ^Determine the radian measure of 

an angle and explain how radian 
measurement is related to a circle of 
radius 1 

6 

26 

0 

0 

0 

2.3.f. *Use trigonometric formulas such 
as addition and double angle formulas 

9 

37 

0 

0 

0 

2.3.g. *Use the law of cosines and the law 

of sines to find unknown sides and angles 
of a triangle 

5 

21 

0 

0 

0 


Note. An asterisk (*) denotes objectives that describe mathematics content beyond that typically taught in a standard 3- 
year course of study (the equivalent of I year of geometry and 2 years of algebra). Therefore, these objectives will be 
selected less often than the others for inclusion on the NAEP assessments. 




KSAs in the Geometry subject area of the NAEP 
Mathematics framework were infrequently found to 
be prerequisite. Two KSAs (3.3.c. and 3.3.d.) were 
determined to be prerequisite in roughly a third of 
college algebra course packets, and one KSA (3.4.a.) 
was found to be prerequisite in slightly less than half 
of the precalculus/calculus course packets. Five KSAs 
(3.3.c, 3.3. d, 3.3.g, 3.4.a, 3.4.f) were determined to be 
prerequisite within at least one finite mathematics 
course packet. None of the Geometry KSAs were 
considered to be prerequisite for statistics based on 
the course packets in the CCCA sample. See Table 43. 

The subject area of Data Analysis, Statistics, and 
Probability did not include many objectives that were 
considered through this content analysis process to be 
prerequisites for many of the sampled course packets; 


27 of the 32 KSAs were determined to be not 
prerequisite in any course packet sampled. The ability 
to read or interpret graphical or tabular representation 
of data (4.1. a.) was considered a prerequisite KSA 
within almost half of the course packets, although 
slightly less commonly found in college algebra 
packets. Completing a graph or solving a problem 
using data in the graph (4.1.b.) was a KSA found to 
be prerequisite within a few course packets in the 
precalculus/calculus, finite mathematics, and statistics 
course titles. One KSA (4.1. d) was considered 
prerequisite in one college algebra course packet, and 
one KSA (4.1.f) was considered prerequisite in one 
statistics course packet. One KSA (4.2. e) was 
determined to be prerequisite in one 
precalculus/calculus course packet and one finite 
mathematics course packet. See Table 44. 


Table 43. Percentages of Course Packets in Which the Mathematics Objectives Under Geometry Were Considered a 
Prerequisite Across all Courses and by Course Title 


Objective 

Overall 

% 

Precalculus/ 

calculus 

% 

College 

algebra 

% 

Finite 

mathematics 

% 

Statistics 

% 

3. Geometry 

3.1. Dimension and shape 

3.I.C. Give precise mathematical 






descriptions or definitions of geometric 
shapes in the plane and in three- 
dimensional space 

3. 1 .d. Draw or sketch from a written 

0 

0 

0 

0 

0 

description plane figures and planar 
images of three-dimensional figures 

3. 1 .e. Use two-dimensional 

3 

0 

10 

0 

0 

representations of three-dimensional 
objects to visualize and solve problems 

3. 1 .f. Analyze properties of three- 

9 

16 

20 

0 

0 

dimensional figures, including spheres 
and hemispheres 

0 

0 

0 

0 

0 


3.2. Transformation of shapes and preservation of properties 


3. 2. a. Recognize or identify types of 
symmetries (e.g., point, line, rotational, 
or self-congruence) of two- and three- 
dimensional figures 


I 


5 


0 


0 


0 




Table 43. Percentages of Course Packets in Which the Mathematics Objectives Under Geometry Were Considered a 
Prerequisite Across all Courses and by Course Title 


Objective 

Overall 

% 

Precalculus/ 

calculus 

% 

College 

algebra 

% 

Finite 

mathematics 

% 

Statistics 

% 

3.2.b. Give or recognize the precise 
mathematical relationship (e.g., 
congruence, similarity, or orientation) 
between a figure and its image under a 
transformation 

0 

0 

0 

0 

0 

3.2.c. Perform or describe the effect of 
a single transformation on two- and 
three-dimensional geometric shapes 
(reflections across lines of symmetry, 
rotations, translations, and dilations) 

1 

0 

5 

0 

0 

3.2.d. Identify transformations, 
combinations, or subdivisions of shapes 
that preserve the area of two- 
dimensional figures or the volume of 
three-dimensional figures 

1 

0 

5 

0 

0 

3.2.e. Justify relationships of congruence 
and similarity and apply these 
relationships using scaling and 
proportional reasoning 

0 

0 

0 

0 

0 

3.2.g. Perform or describe the effects of 
successive transformations 

1 

5 

0 

0 

0 

3.3. Relationships between geometric figures 






3.3.b Apply geometric properties and 
relationships to solve problems in two 
and three dimensions 

8 

5 

20 

0 

0 

3.3.c. Represent problem situations with 
geometric models to solve mathematical 
or real-world problems 

14 

0 

30 

5 

0 

3.3.d. Use the Pythagorean theorem to 
solve problems in two- or three- 
dimensional situations 

19 

0 

40 

10 

0 

3.3.e. Recall and interpret definitions 
and basic properties of congruent and 
similar triangles, circles, quadrilaterals, 
polygons, parallel, perpendicular and 
intersecting lines, and associated angle 
relationships 

3 

0 

5 

0 

0 

3.3.f. Analyze properties or relationships 
of triangles, quadrilaterals, and other 
polygonal plane figures 

0 

0 

0 

0 

0 

3.3.g. Analyze properties and 
relationships of parallel, perpendicular, 
or intersecting lines including the angle 

3 

5 

0 

5 

0 


relationships that arise in these cases 




Table 43. Percentages of Course Packets in Which the Mathematics Objectives Under Geometry Were Considered a 
Prerequisite Across all Courses and by Course Title 


Objective 

Overall 

% 

Precalculus/ 

calculus 

% 

College 

algebra 

% 

Finite 

mathematics 

% 

Statistics 

% 

3.3.h. Analyze properties of circles and 
the intersections of lines and circles 
(inscribed angles, central angles, 
tangents, secants, and chords) 

0 

5 

0 

0 

0 

3.4. Position, direction, and coordinate geometry 






3.4. a Solve problems involving the 
coordinate plane such as the distance 
between two points, the midpoint of a 
segment, or slopes of perpendicular or 
parallel lines 

16 

42 

15 

10 

0 

3.4. b Describe the intersections of lines 
in the plane and in space, of a line and a 
plane, or of two planes in space 

1 

0 

5 

0 

0 

3.4.c Describe or identify conic sections 
and other cross sections of solids 

5 

21 

0 

0 

0 

3.4. d Represent two-dimensional figures 
algebraically using coordinates and/or 
equations 

8 

21 

10 

0 

0 

3.4. e *Use vectors to represent velocity 
and direction; multiply a vector by a 
scalar and add vectors both algebraically 
and graphically 

0 

0 

0 

0 

0 

3.4.f Find an equation of a circle given its 
center and radius and, given an equation 
of a circle, find its center and radius 

8 

21 

5 

5 

0 

3.4.g *Graph ellipses and hyperbolas 
whose axes are parallel to the 
coordinate axes and demonstrate 
understanding of the relationship 
between their standard algebraic form 
and their graphical characteristics 

3 

1 1 

0 

0 

0 

3.4.h ^Represent situations and solve 
problems involving polar coordinates 

0 

0 

0 

0 

0 

3.5. Mathematical reasoning in geometry 






3. 5. a Make, test, and validate geometric 
conjectures using a variety of methods 
including deductive reasoning and 
counterexamples 

0 

0 

0 

0 

0 

3.5.b Determine the role of hypotheses, 
logical implications, and conclusion in 

0 

0 

0 

0 

0 


proofs of geometric theorems 




Table 43. Percentages of Course Packets in Which the Mathematics Objectives Under Geometry Were Considered a 
Prerequisite Across all Courses and by Course Title 


Objective 

Overall 

% 

Precalculus/ 

calculus 

% 

College 

algebra 

% 

Finite 

mathematics 

% 

Statistics 

% 

3.5.c Analyze or explain a geometric 
argument by contradiction 

0 

0 

0 

0 

0 

3.5.d Analyze or explain a geometric 
proof of the Pythagorean theorem 

0 

0 

0 

0 

0 

3.5.e Prove basic theorems about 

congruent and similar triangles and 
circles 

0 

0 

0 

0 

0 


Note. An asterisk (*) denotes objectives that describe mathematics content beyond that typically taught in a standard 3- 
year course of study (the equivalent of I year of geometry and 2 years of algebra). Therefore, these objectives will be 
selected less often than the others for inclusion on the NAEP assessments. 


Table 44. Percentages of Course Packets in Which the Mathematics Objectives Under Data Analysis, Statistics, and 

Probability Were Considered a Prerequisite Across all Courses and by Course Title 

Objective 

Precalculus/ 

Overall calculus 

% % 

College 

algebra 

% 

Finite 

mathematics Statistics 

% % 

4. Data analysis, statistics, and probability 

4. 1 . Data representation 





4. 1 .a. Read or interpret graphical or 
tabular representations of data 

49 58 

35 

55 

50 

4. 1 .b. For a given set of data, complete a 
graph and solve a problem using the 
data in the graph (histograms, 
scatterplots, and line graphs) 

8 21 

0 

5 

5 

4. 1.c. Solve problems involving 
univariate or bivariate data 

0 0 

0 

0 

0 

4. 1 .d. Given a graphical or tabular 
representation of a set of data, 
determine whether information is 
represented effectively and 
appropriately 

1 0 

5 

0 

0 

4. 1 .e. Compare and contrast different 
graphical representations of univariate 
and bivariate data 

0 0 

0 

0 

0 

4. 1 .f. Organize and display data in a 
spreadsheet in order to recognize 
patterns and solve problems 

1 0 

0 

0 

5 






Table 44. Percentages of Course Packets in Which the Mathematics Objectives Under Data Analysis, Statistics, and 
Probability Were Considered a Prerequisite Across all Courses and by Course Title 


Objective 

Overall 

% 

Precalculus/ 

calculus 

% 

College 

algebra 

% 

Finite 

mathematics 

% 

Statistics 

% 

4.2. Characteristics of data sets 






4. 2. a. Calculate, interpret, or use 
summary statistics for distributions of 
data including measures of typical value 
(mean and median), position (quartiles 
and percentiles), and spread (range, 
interquartile range, variance, and 
standard deviation) 

0 

0 

0 

0 

0 

4.2.b. Recognize how linear 
transformations of one-variable data 
affect mean, median, mode, range, 
interquartile range, and standard 
deviation 

0 

0 

0 

0 

0 

4.2.c. Determine the effect of outliers 
on mean, median, mode, range, 
interquartile range, or standard 
deviation 

0 

0 

0 

0 

0 

4.2.d. Compare data sets using summary 
statistics (mean, median, mode, range, 
interquartile range, or standard 
deviation) describing the same 
characteristic for two different 
populations or subsets of the same 
population 

0 

0 

0 

0 

0 

4.2.e. Approximate a trend line if a 
linear pattern is apparent in a 
scatterplot or use a graphing calculator 
to determine a least-squares regression 
line and use the line or equation to 
make predictions 

3 

5 

0 

5 

0 

4.2.f. Recognize that the correlation 
coefficient is a number from - 1 to + 1 
that measures the strength of the linear 
relationship between two variables; 
visually estimate the correlation 
coefficient (e.g., positive or negative, 
closer to 0, .5, or 1 .0) of a scatterplot 

0 

0 

0 

0 

0 

4.2.g. Know and interpret the key 
characteristics of a normal distribution 
such as shape, center (mean), and 
spread (standard deviation) 

0 

0 

0 

0 

0 

4.3. Experiments and samples 






4. 3. a. Identify possible sources of bias in 
sample surveys and describe how such 

0 

0 

0 

0 

0 


bias can be controlled and reduced 




Table 44. Percentages of Course Packets in Which the Mathematics Objectives Under Data Analysis, Statistics, and 
Probability Were Considered a Prerequisite Across all Courses and by Course Title 


Objective 

Overall 

% 

Precalculus/ 

calculus 

% 

College 

algebra 

% 

Finite 

mathematics 

% 

Statistics 

% 

4.3.b. Recognize and describe a method 
to select a simple random sample 

0 

0 

0 

0 

0 

4.3.c. *Draw inferences from samples, 
such as estimates of proportions in a 
population, estimates of population 
means, or decisions about differences in 
means for two "treatments" 

0 

0 

0 

0 

0 

4.3.d. Identify or evaluate the 
characteristics of a good survey or of a 
well-designed experiment 

0 

0 

0 

0 

0 

4.3.e. ^Recognize the differences in 
design and in conclusions between 
randomized experiments and 
observational studies 

0 

0 

0 

0 

0 

4.4. Probability 






4.4. a. Recognize whether two events 
are independent or dependent 

0 

0 

0 

0 

0 

4.4. b. Determine the theoretical 
probability of simple and compound 
events in familiar or unfamiliar contexts 

0 

0 

0 

0 

0 

4.4.c. Given the results of an 
experiment or simulation, estimate the 
probability of simple or compound 
events in familiar or unfamiliar contexts 

0 

0 

0 

0 

0 

4.4.d. Use theoretical probability to 
evaluate or predict experimental 
outcomes 

0 

0 

0 

0 

0 

4.4. e. Determine the number of ways an 
event can occur using tree diagrams, 
formulas for combinations and 
permutations, or other counting 
techniques 

0 

0 

0 

0 

0 

4.4. h. Determine the probability of 
independent and dependent events 

0 

0 

0 

0 

0 

4.4.i. Determine conditional probability 
using two-way tables 

0 

0 

0 

0 

0 

4.4.j. Interpret and apply probability 
concepts to practical situations 

0 

0 

0 

0 

0 

4.4. k. *Use the binomial theorem to 
solve problems 

0 

0 

0 

0 

0 




Table 44. Percentages of Course Packets in Which the Mathematics Objectives Under Data Analysis, Statistics, and 
Probability Were Considered a Prerequisite Across all Courses and by Course Title 




Precalculus/ 

College 

Finite 



Overall 

calculus 

algebra 

mathematics 

Statistics 

Objective 

% 

% 

% 

% 

% 

4.5. Mathematical reasoning with data 






4. 5. a. Identify misleading uses of data in 
real-world settings and critique different 
ways of presenting and using 
information 

0 

0 

0 

0 

0 

4.5.b. Distinguish relevant from 
irrelevant information, identify missing 
information, and either find what is 
needed or make appropriate 
approximations 

0 

0 

0 

0 

0 

4.5.c. *Recognize, use, and distinguish 
between the processes of mathematical 
(deterministic) and statistical modeling 

0 

0 

0 

0 

0 

4.5.d. Recognize when arguments based 
on data confuse correlation with 

0 

0 

0 

0 

0 

causation 






4.5.e. ^Recognize and explain the 
potential errors caused by extrapolating 
from data 

0 

0 

0 

0 

0 


Note. An asterisk (*) denotes objectives that describe mathematics content beyond that typically taught in a standard 3-year 
course of study (the equivalent of I year of geometry and 2 years of algebra). Therefore, these objectives will be selected less 
often than the others for inclusion on the NAEP assessments. 


Few of the KSAs represented by objectives in the 
Algebra subject area of the NAEP Mathematics 
framework were considered to be prerequisite 
consistently within or across course titles. Four of the 
30 KSAs (5.3.c, 5.3.e, 5.3.f, 5 .4. a) were determined to 
be prerequisite in at least one packet from all course 
tides. Eleven KSAs (5.1.e, 5.1.g, 5.1.h, 5.h.i, 5.2.a, 5.2.e, 
5.3.b, 5.3.d, 5.4.c, 5.4.e, 5.4.f) were determined to be 
prerequisite in at least one packet from three course 
tides. Flowever, across course titles, the percentages of 
course packets where the KSA was deemed 
prerequisite were not consistent. For example, the 
ability to evaluate algebraic expressions including 
polynomials and rational expressions (5.3.e) was found 
to be prerequisite in as many as 90% of course packets 


within one course title (college algebra) and as little as 
1 0% of course packets within another course title 
(statistics). This was also the KSA that was most 
frequently identified as prerequisite across course titles. 

Similarly to the Number Properties and Operations subject 
area, there were generally more KSAs considered 
prerequisite in precalculus/ calculus and college algebra 
than in finite mathematics or statistics. For any given 
KSA under the Algebra subject area, the KSA was 
considered prerequisite in fewer than 50% of the finite 
mathematics or statistics course packets. Only three 
Algebra KSAs (5.3.g, 5.4.g, 5.5.b) were determined not to 
be prerequisite in any course packet across all course 
titles in the CCCA sample. See Table 45. 




Table 45. Percentages of Course Packets in Which the Mathematics Objectives Under Algebra Were Considered a 
Prerequisite Across all Courses and by Course Title 


Objective 

Overall 

% 

Precalculus/ 

calculus 

% 

College 

algebra 

% 

Finite 

mathematics 

% 

Statistics 

% 

5. Algebra 






5. 1 . Patterns, relations, and functions 






5.1. a. Recognize, describe, or extend 
numerical patterns, including arithmetic 
and geometric progressions 

5 

16 

0 

5 

0 

5.l.b. Express linear and exponential 
functions in recursive and explicit form 
given a table, verbal description, or 
some terms of a sequence 

5 

16 

0 

5 

0 

5. 1 .e. Identify or analyze distinguishing 
properties of linear, quadratic, rational, 
exponential, or trigonometric 
functions from tables, graphs, or 
equations 

’ll 

68 

30 

10 

0 

5. 1 .g. Determine whether a relation, 
given in verbal, symbolic, tabular, or 
graphical form, is a function 

16 

42 

20 

5 

0 

5.l.h. Recognize and analyze the 
general forms of linear, quadratic, 
rational, exponential, or trigonometric 
functions 

20 

53 

20 

10 

0 

5. 1 .i. *Determine the domain and range 
of functions given in various forms and 
contexts 

25 

68 

30 

5 

0 

5. 1 .j. Given a function, determine its 
inverse if it exists and explain the 
contextual meaning of the inverse for a 
given situation 

8 

32 

0 

0 

0 

5.2. Algebraic representations 






5. 2. a. Create and translate between 
different representations of algebraic 
expressions, equations, and inequalities 
(e.g., linear, quadratic, exponential, or 
trigonometric) using symbols, graphs, 
tables, diagrams, or written 
descriptions 

44 

89 

70 

20 

0 

5.2.b. Analyze or interpret relationships 
expressed in symbols, graphs, tables, 
diagrams (including Venn diagrams), or 
written descriptions and evaluate the 

8 

21 

0 

10 

0 


relative advantages or disadvantages of 
different representations to answer 
specific questions 




Table 45. Percentages of Course Packets in Which the Mathematics Objectives Under Algebra Were Considered a 
Prerequisite Across all Courses and by Course Title 


Objective 

Overall 

% 

Precalculus/ 

calculus 

% 

College 

algebra 

% 

Finite 

mathematics 

% 

Statistics 

% 

5.2.d. Perform or interpret 
transformations on the graphs of linear, 
quadratic, exponential, and 
trigonometric functions 

10 

37 

5 

0 

0 

5.2.e. Make inferences or predictions 
using an algebraic model of a situation 

19 

32 

25 

20 

0 

5.2.f. Given a real-world situation, 
determine if a linear, quadratic, rational, 
exponential, logarithmic, or 
trigonometric function fits the 
situation 

8 

21 

10 

0 

0 

5.2.g. Solve problems involving 
exponential growth and decay 

6 

26 

0 

0 

0 

5.2.h. *Analyze properties of 
exponential, logarithmic, and rational 
functions 

5 

21 

0 

0 

0 

5.3. Variables, expressions, and operations 






5.3.b. Write algebraic expressions, 
equations, or inequalities to represent a 
situation 

46 

68 

75 

40 

0 

5.3.c. Perform basic operations, using 
appropriate tools, on algebraic 
expressions including polynomial and 
rational expressions 

53 

89 

90 

30 

5 

5.3.d. Write equivalent forms of 
algebraic expressions, equations, or 
inequalities to represent and explain 
mathematical relationships 

29 

63 

30 

25 

0 

5.3.e. Evaluate algebraic expressions 
including polynomials and rational 
expressions 

58 

89 

90 

45 

10 

5.3.f. Use function notation to evaluate 
a function at a specified point in its 
domain and combine functions by 
addition, subtraction, multiplication, 
division, and composition 

23 

68 

5 

15 

5 

5.3.g. ^Determine the sum of finite and 
infinite arithmetic and geometric series 

0 

0 

0 

0 

0 

5.3.h. Use basic properties of 
exponents and logarithms to solve 
problems 

28 

42 

70 

0 

0 




Table 45. Percentages of Course Packets in Which the Mathematics Objectives Under Algebra Were Considered a 
Prerequisite Across all Courses and by Course Title 


Objective 

Overall 

% 

Precalculus/ 

calculus 

% 

College 

algebra 

% 

Finite 

mathematics 

% 

Statistics 

% 

5.4. Equations and inequalities 






5.4. a. Solve linear, rational, or quadratic 
equations or inequalities, including 
those involving absolute value 

48 

89 

75 

25 

5 

5.4. c. Analyze situations, develop 
mathematical models, or solve 
problems using linear, quadratic, 
exponential, or logarithmic equations 
or inequalities symbolically or 
graphically 

35 

89 

50 

5 

0 

5.4. d. Solve (symbolically or graphically) 
a system of equations or inequalities 
and recognize the relationship between 
the analytical solution and graphical 
solution 

1 

0 

0 

5 

0 

5.4. e. Solve problems involving special 
formulas such as: A = P(l + r)\ A = Pe n 

10 

26 

10 

5 

0 

5.4.f. Solve an equation or formula 
involving several variables for one 
variable in terms of the others 

15 

26 

25 

10 

0 

5.4.g. Solve quadratic equations with 
complex roots 

0 

0 

0 

0 

0 

5.5. Mathematical reasoning in algebra 






5. 5. a. Use algebraic properties to 
develop a valid mathematical argument 

3 

5 

0 

5 

0 

5.5.b. Determine the role of 
hypotheses, logical implications, and 
conclusions in algebraic argument 

0 

0 

0 

0 

0 

5.5.c. Explain the use of relational 
conjunctions (and, or) in algebraic 
arguments 

1 

0 

0 

5 

0 


Note. An asterisk (*) denotes objectives that describe mathematics content beyond that typically taught in a standard 3-year 
course of study (the equivalent of I year of geometry and 2 years of algebra). Therefore, these objectives will be selected less 
often than the others for inclusion on the NAEP assessments 


An examination of the 1 5 prerequisite KSAs identified 
by the content reviewers that are not represented by 
objectives in the NAEP framework reveals that few of 
the non-NAEP additional KSAs were considered to be 
common prerequisites across course packets. Eight 
non-NAEP prerequisite KSAs were identified in nine 


or more packets. Three of these relate to the use of 
calculators, two more broadly to the use of technology 
and computers, one involves using interval notation 
and another set notation, and one incorporates reading 
skill into the ability to convert sentences into 
mathematical expressions. The remaining non-NAEP 




prerequisite KSAs were found in six or fewer course 
packets. Notably, despite condensing the prerequisite 
KSAs when redundancy was clearly evident, the 1 5 
non-NAEP additional KSAs do not represent 1 5 
unique and unrelated abilities. KSAs related to the use 
of technology, including web-based content, 
computers, and calculators, comprise one third of the 
total non-NAEP additional KSAs. The ability to read 
and communicate mathematical information 
comprised four of the non-NAEP additional KSAs. 
Two relate to the Cartesian coordinate system and 
determining the equation of a line. See Table 46. 


Prerequisite KSA Narrative Descriptions 
The primary objective of the NAEP expert review of 
the group review results was to develop a more 
integrative narrative description of the knowledge, 
skills, and abilities a student would need to possess on 
entry into credit-bearing college courses. The 
narrative descriptions contain the key prerequisite 
content and skills from the knowledge, skills, and 
abilities contained in the NAEP-specific content 
maps, KSA exclusions, and non-NAEP additional 
KSAs identified by the content review groups. 


Table 46. Non-NAEP Additional KSAs Determined to Be Prerequisite for Mathematics Courses 


Non-NAEP additional prerequisite KSAs 

Number of 

course 

packets 

Precalculus/ 

calculus 

College 

algebra 

Finite 

mathematics 

Statistics 

Reading comprehension sufficient to convert 
sentences to mathematical expressions 

20 

5 

5 

5 

5 

Use of technology 

19 

4 

5 

5 

5 

Basic calculator skills 

15 

2 

4 

5 

4 

Use of scientific calculator 

14 

2 

3 

5 

4 

Understanding and using interval notation 

12 

6 

6 

0 

0 

Facility with/use of graphing calculators 

Facility in use of computers, including loading 

9 

2 

2 

2 

3 

software, submitting homework, taking quizzes 
online, and manipulating charts 

Understanding and using basic set notation, 

9 

0 

3 

0 

6 

including symbols for element of, not an 
element of, subset of, and not a subset of 

9 

2 

4 

2 

1 

Writing ability sufficient to prepare a brief 
report and a reflection paper 

6 

2 

0 

1 

3 

Communicating mathematics in written form 

Ability to: given points, determine the equation 
of a line; given a point and slope, determine the 

5 

0 

0 

0 

5 

equation of a line; given an equation and a point, 
determine the equation of a line parallel and 
perpendicular to the given equation 

5 

4 

0 

1 

0 

Ability to work in groups 

5 

1 

0 

2 

2 

Knowledge of the Cartesian coordinate system 

1 

1 

0 

0 

0 

Problem-solving skills 

1 

1 

0 

0 

0 




The NAEP reading experts generated a single 
description of prerequisite KSAs, integrating findings 
across all courses in the study with substantial reading 
demands. The NAEP mathematics experts 
determined that mathematics courses differed from 
each other enough to make a single overall 
description of mathematics prerequisite KSAs 
problematic. The mathematics team addressed this 
issue by identifying a set of overall prerequisite KSAs 
plus course-specific descriptions of prerequisite 
knowledge, skills, and abilities. 

Following are the summaries describing the 
prerequisite KSAs that were evident in the analyzed 
courses. Because these descriptions account for the 
KSA exclusions and include the non-NAEP 
additional KSAs, they are not a listing of NAEP 
objectives, nor are they NAEP-specific or only 
interpretable in the context of NAEP. Underlined 
statements indicate non-NAEP additional prerequisite 
KSAs not contained in the content maps. 


Prerequisite KSAs Compared to Content 
Addressed in NAEP 2009 and 2013 Item Pools 

The NAEP experts used the narrative descriptions of 
prerequisites KSAs as reference points for reviewing 
item pools from the 2009 and 2013 NAEP 
administrations. In general, most of the reading items 
seemed to align with the CCCA prerequisite KSAs 
found for entry-level courses requiring substantial 
reading. After reviewing the item pools against the 
narrative descriptions, the NAEP experts concluded 
that the prerequisite KSAs identified in the CCCA 
study assume students have prior knowledge of the 
content (e.g., a student in a U.S. government course 
has some prior knowledge of government to 
comprehend a passage) whereas the NAEP reading 
selections and associated items appeared to be 
selected for wide accessibility, i.e., not geared towards 
students with prior knowledge. 

For mathematics, the narrative description of 
prerequisite KSAs focused more heavily on 


NAEP Reading Expert Narrative Description of Prerequisite KSAs 

To be minimally prepared for college reading of printed texts and online resources, students need to be able to locate and recall 
specific information such as definitions, facts, events, and other supporting details and to determine word meanings as used in 
context. Students should be able to distinguish between primary and secondary materials and fact and opinion. 

Students need to be able to identify and describe causal relationships and organizational structure of texts, such as 
comparison/contrast, problem/solution, and enumeration. Within and across texts they need to be able to make simple and 
complex inferences, summarize major themes and ideas, draw conclusions, and provide supporting information. Students need to 
be able to describe and analyze how an author uses text features such as graphics, tables, and charts to convey meaning. 

Students need to be able to take different perspectives in relation to a text and analyze, critique, and evaluate author’s purpose, 
point of view, and bias. Students need to be able to find evidence in support of an argument and evaluate the strength and 
quality of the evidence, determine the quality of counter arguments, and judge the coherence and logic of an argument. Students 
need to be able to critique and evaluate the significance of the historical and cultural context of texts. Finally, students need to be 

able to apply information from readings to contemporary issues and real-world problems. 

In addition to these knowledge, skills, and abilities, to be minimally prepared for literature courses students need to be able to 
describe and analyze relations among settings, characters, and themes in a variety of literary texts. Students need to be able to 
interpret the text critically, evaluating the role of figurative language and literary devices and describing, analyzing, and evaluating 
characters’ conflicts, motivations, and decisions. 

Note: Technology literacy was identified in some artifacts as being prerequisite. “Digital” text is included here, but not addressed. 


NAEP Mathematics Expert Narrative Description of Prerequisite KSAs 

Number Properties and Operations: For all four courses, students should be able to represent, interpret, and solve problems 
with real numbers. They can compare and order numbers as well as use conventions about the order of numerical operations. 
Integral or simple fractional powers of real numbers are especially needed for college algebra and precalculus/calculus. For none 
of the courses do students need to know about logarithms, but knowledge of exponents is needed for college algebra and 
precalculus/calculus. Students entering college algebra or precalculus/calculus courses need to be able to solve application 
problems involving factors and multiples. They also need to be able to represent, interpret, compare, or perform operations with 
expressions involving absolute values. 

Measurement and Geometry: Overall, none of the objectives [NAEP-specific KSAs] in measurement or geometry was deemed 
a prerequisite other than some emphasis on coordinate geometry for the study of precalculus/calculus, and some attention to 
area, perimeter, and the Pythagorean Theorem for college algebra. 

Statistics: For all four courses, students should be able to read or interpret graphs or tables. Otherwise, none of the statistics 
objectives was deemed a prerequisite. 

Algebra: For statistics, none of the algebra objectives is a prerequisite. For finite mathematics, a minimal knowledge of algebra is 
needed, such as writing and evaluating expressions. College algebra and precalculus/calculus have as a prerequisite the ability to 
write and translate between different representations using algebraic expressions, equations, and inequalities. They also require 
that students be able to evaluate and perform basic operations on expressions. In addition, students entering those courses 
should be able to analyze, or solve problems with, linear, rational, or quadratic equations or inequalities. For precalculus/calculus, 
students should be able to write equivalent forms of algebraic expressions, equations, or inequalities to represent situations. 
Precalculus/calculus students should also have knowledge of functions, including domain and range. They should be able to 
evaluate functions and analyze function properties. 

Non-NAEP Additional KSAs. Overall, students should be familiar with sets and their properties and notation. For their courses, 
they should have facility in the use of computers and calculators. 


application of mathematics than is evident in the 
NAEP item pools. The items are skill-based and 
procedural in nature rather than applied. The CCCA 
content contained more instances of application 
problems. This finding is consistent with the NAEP 
framework, but the NAEP items do not reflect this to 
the same degree. Another significant difference 
between NAEP mathematics and the narrative 
description of prerequisite KSAs is the absence of 
measurement and geometry in the majority of courses 
and the presence of measurement and geometry 
content in the NAEP item pools. 

College-Course Prerequisites Compared to Other 
Study Findings 

The narrative descriptions from this study were 
compared to the borderline performance descriptions 


(BPDs) for college preparedness (version 4.0) from the 
JSS study. The BPDs and the narrative descriptions 
from this study were compared for breadth of KSAs 
included and the level of cognitive demand (Bloom, 
1956) represented within those KSAs. Appendix W 
contains the BPDs from the JSS study. 

Comparison of reading JSS BPD and CCCA 
narrative description. A comparison of the JSS BPD 
and the CCCA narrative description for reading 
suggests that the level of cognitive demand and 
breadth of prerequisite KSAs are greater in the CCCA 
study than in the JSS study. Almost all KSAs 
described in the JSS BPD are included in the CCCA 
narrative description; one exception is that the JSS 
BPD includes students’ ability to explain, when 
locating relevant information in texts, why specific 


information is included. This ability to explain why an 
author includes certain information is implied within 
the CCCA narrative description under the ability to 
analyze, evaluate, and critique author’s purpose, and 
to locate and recall supporting details. 

The JSS BPD includes few verbs, such as critique or 
evaluate, that relate to higher-order thinking skills. 
More common in the JSS BPD are verbs such as 
locate, identify, understand, interpret, and explain. 

The CCCA narrative description includes more KSAs 
that represent demands for higher-order skills. As an 
example of the distinction, the JSS BPD includes the 
KSA, “They should be able to identify or explain the 
author’s point or central theme and supporting textual 
evidence.” The statement in the CCCA narrative 
description that references similar KSAs relating to 
author’s purpose is more complex: “Students need to 
be able to take different perspectives in relation to a 
text and analyze, critique, and evaluate author’s 
purpose, point of view, and bias.” 

The CCCA narrative description includes several 
KSAs that are not evident in the JSS BPD. The JSS 
BPD makes no reference to KSAs related to 
evaluating the strength and quality of evidence in 
support of an argument, the quality of 
counterarguments, and the overall coherence and 
logic of an argument. The CCCA narrative 
description also draws in the need for students to be 
able to understand the cultural and historical contexts 
of texts, and to be able to apply information from 
texts to contemporary issues and real-world problems. 
Also in the CCCA narrative description but not the 
JSS BPD is the specification that the prerequisite 
KSAs apply to the reading of both printed and online 


resources. A final distinction between the CCCA 
narrative description and the JSS BPD is the inclusion 
in the CCCA narrative description of a reference to 
students’ ability to distinguish between primary and 
secondary materials and fact and opinion. 

Comparison of mathematics JSS BPD and CCCA 
narrative description. The JSS BPD and CCCA 
mathematics description are more challenging to 
compare than the JSS reading BPD to the narrative 
description for the courses with substantial reading 
demands. The JSS BPD and CCCA narrative 
description include many similar prerequisite KSAs; 
however, there are some prerequisite KSAs unique to 
either the JSS BPD or CCCA narrative description. 
The JSS BPD places more emphasis than the CCCA 
narrative description on surface area, volume, and 
other units of measurement in the Measurement and 
Geometry subject areas. Likewise, the CCCA narrative 
description states that students need only read or 
interpret graphs and tables in the Data Analysis, 
Probability and Statistics subject area, whereas the JSS 
BPD emphasizes computing mean, median, mode, 
and other basic concepts of probability. There is 
reasonable overlap in the Number Properties and 
Operations and Algebra subject areas between the JSS 
BPD and the CCCA narrative description. Lastly, the 
CCCA narrative description states that students 
should be familiar with sets and their properties and 
notation and should have facility in the use of 
computers and calculators. The JSS BPD includes 
neither of these prerequisite KSAs. However, the JSS 
BPD was drawn entirely from the NAEP framework 
objectives, so the exclusion of any KSA that was not 
based on a NAEP objective is to be expected. 


CONCLUSIONS 


The College Course Content Analysis study, using a 
convergent consensus process involving expert 
judgment, systematic review processes, and 
standardized sets of course artifacts, complements 
other types of studies that have been sponsored by 
the National Assessment Governing Board in its 12th 
Grade Preparedness Research Program. The CCCA 
study was designed to elicit, through an evidence- 
based content analysis design, the knowledge, skills, 
and abilities that students need to bring with them 
into entry-level, credit-bearing college courses. The 
four research questions that formed the focus of this 
study are addressed in detail below. 

Research Question I 

What are the prerequisite knowledge, skills, a nd abilities 
(hereafter referred to as prerequisite KSAs) in mathematics and 
reading to qualify for entry-level, credit-bearing courses that 
satisfy general education requirements? The. findings from 
the study suggest that there is a set of reading KSAs 
that are prerequisite to entering entry-level, credit- 
bearing college courses that have substantive 
informational text reading demands, and a set of 
prerequisite KSAs for courses that demand 
engagement with literary texts. The findings for 
mathematics were specific to a course title, without a 
single set of prerequisite KSAs that covered all entry- 
level, credit-bearing courses. Most of the prerequisite 
KSAs for the sample of both mathematics course 
titles and course titles that require extensive college- 
level reading are reflected in the NAEP frameworks. 
However, there were non-NAEP additional KSAs 
that are considered prerequisite for these courses that 
are not included in the NAEP frameworks. 


Reading Prerequisite KSAs 

The KSAs that were prerequisite to entry-level, credit- 
bearing college courses that have substantial reading 
demands included those requiring various levels of 
cognitive demand. The KSAs that appear to be 
ubiquitous in their relevance as prerequisite to college 
course entry were represented within the NAEP 
framework cognitive domain locate/ Recall: Locate or 
recall textuallj explicit information within and across texts, 
which may involve making simple inferences as needed for literal 
comprehension. Slightly less uniform but still very 
commonly found in the packets were KSAs that are 
reflected in the NAEP framework’s cognitive domain 
Integrate / Interpret: Make complex inferences within and across 
texts (e.g., compare or connect ideas, perspectives, 
problems, or situations). This is consistent with the 
information provided in the course syllabi, texts, 
assignments, and assessments. KSAs that relate to 
locating or recalling information or integrating and 
interpreting information are foundational to being 
able to succeed at work that demands deeper levels of 
cognitive complexity, which are likely taught in 
college-level courses. 

There are few differences in prerequisites among the 
courses that incorporate primarily informational texts, 
suggesting that the list in the Results section may well 
represent the prerequisite KSAs for such entry-level 
courses. English literature was the sole course title 
that relies primarily on literary texts. Within the KSAs 
represented in the NAEP framework, there were few 
differences in prerequisites if one allows for expected 
differences based on whether standards refer to 
informational or literary texts. A slight difference, 
however, is in KSAs related to a student’s ability to 
critique or evaluate texts; related KSAs are more 




commonly considered to be prerequisite in the literary 
text-dependent course than in the informational text- 
dependent courses. 

Referencing the course artifacts, it was evident to the 
content reviewers that students are asked to engage 
with texts in sophisticated ways. This comes through 
in the KSAs that were determined to be prerequisite 
but were not a part of the NAEP framework. These 
non-NAEP additional KSAs included the ability to 
critique/ evaluate the significance of the historical or 
cultural context of a text or author, analyze both 
factual and theoretical information in informational 
text, and discern credibility of a source. Within the 
NAEP framework, the objective “take different 
perspectives in relation to a text” also illustrates this 
conclusion, and was considered a prerequisite KSA 
across all course titles. 

Facility with technology and accessing online 
resources also arose as a prerequisite KSA for courses 
with substantial reading demands. 

Mathematics Prerequisite KSAs 
The majority of the KSAs that were determined to be 
prerequisite to all the entry-level college mathematics 
courses included in this study related to objectives in 
the NAEP framework under the subject area Number 
and Operations. The majority of mathematics course 
prerequisites, across all courses, were represented 
within the NAEP framework’s Number and Operations 
subject area, specifically the Number sense, Properties of 
number and operations , and Number operation subtopics. 
Prerequisite KSAs for precalculus/calculus and 
college algebra were notably different from the 
prerequisite KSAs for finite mathematics and 
statistics. Prerequisite KSAs for precalculus /calculus 
and college algebra were represented by the objectives 
within the Variables, Expressions, and Operations and 
Equations and Inequalities subtopics of the Algebra 
subject area, whereas there were fewer prerequisite 


KSAs from these subject areas for statistics and finite 
mathematics. KSAs related to most of the objectives 
in the NAEP framework subject areas of Measurement; 
Data Analysis, Statistics, and Probability; and Geometry 
were not commonly determined to be prerequisite to 
the course tides sampled. However, many KSAs were 
found in a minority of course packets, which suggests 
that, although these KSAs may not be consistently 
prerequisite to entry into these courses, there is a 
range of prerequisites across course tides and within 
course titles as they are taught at different institutions. 

Several non-NAEP additional KSAs that were not 
found within the NAEP framework were identified as 
prerequisites for mathematics courses. The most 
commonly noted non-NAEP additional KSAs that 
were found within the artifacts related to use of 
technology (e.g., calculators, online resources) and the 
ability to read and communicate in various modes 
about mathematics. 

Research Question 2 

Plow do these prerequisite KSAs compare with the 2009 and 
201 3 NAEP mathematics and readingframeworks and item 
pools? The NAEP reading items pools measure 
content that corresponds fairly well with the 
prerequisite KSAs developed in this study. The text 
excerpts in the 2009 and 2013 NAEP reading item 
pools did not cover the full range of texts necessary 
to assess all of the KSAs. This limits the ability of the 
12th grade reading assessment to capture fully the 
degree of student preparedness for college courses 
that require significant amounts of informational text. 
The NAEP items focus more on the density and 
complexity of expository, discipline-based text. One 
explanation of this distinction is that NAEP reading 
passages are selected based on the criterion that 
students do not need to have significant prior content 
knowledge in order to be able to comprehend the 
passage and answer the questions associated with it. 


The NAEP reading framework aligned well with the 
prerequisite KSAs identified in this study, especially in 
the cognitive domains of Ijocate / Recall and 
Integrate / Interpret. The prerequisite KSAs reflected less 
evidence of the need to be able to critique or evaluate 
when entering the course. This may reflect the fact that 
these skills are taught and developed more intensively 
within the college course and, therefore, are not viewed 
as a prerequisite. The non-NAEP additional KSAs that 
went beyond the NAEP framework did, however, 
address the ability to critique and evaluate. The 
prerequisite KSAs also included the application of 
knowledge to the real world and the interpretation of 
information through historical and cultural lenses with 
close attention to evaluating an author’s purpose and 
potential biases in text. 

One challenge that emerged from comparing entry- 
level college course content to the 12th grade NAEP 
mathematics framework is that many college courses 
require or imply mastery of mathematical knowledge 
and skills that are found in the 8th grade objectives 
and, occasionally, the 4th grade objectives. Some 
additional prerequisite KSAs were implied within the 
NAEP frameworks, and others were explicitly stated in 
the frameworks for earlier grade levels, but these earlier 
grade frameworks were not referenced by NAEP 
experts as they considered the results of this study. 

Because the 12th grade framework focuses on the 
mathematics that is prerequisite mainly to college 
algebra or calculus courses, it does not identify 
content needed to be prepared for some other 
introductory college mathematics courses, including 
some examined in this study. The framework 
currently identifies the mathematics that all 12th grade 
students need to know to pursue a wide variety of 
potential options after high school. The fact that 
many of the courses in this study include KSAs that 
are addressed in the frameworks at lower grades 
suggests the importance of ensuring that students 


have thoroughly mastered foundational mathematics 
content and concepts taught in earlier grades. 

Whereas the 4th and 8th grade objectives may identify 
the content and NAEP may test basic understanding, 
by the time students enter college they need to have 
mastered this content and these skills at an automatic 
level. They need strong fluency with the foundational 
mathematics because the courses examined in this 
study do not teach this content as new material, nor 
in many cases do they spend much time, if any, on 
review. Instead, instmctors may assume incoming 
students are fully capable of employing mathematics 
they first learned in earlier grades. Student success 
often hedges on this assumption being true, given 
that college instructors do not reteach this material. 

Course materials suggest that mathematics in entry- 
level courses appears applied in nature, with an 
emphasis on real-world problems and mathematical 
modeling. The entry-level courses analyzed in this 
study reflect a wider range of uses of mathematics. 
They are in many cases designed to lead to subsequent 
courses that begin to develop student ability to apply 
the mathematics in specific contexts dictated by the 
nature of the discipline. High school courses, by 
contrast, are designed to prepare students for a wide 
range of potential college mathematics courses or are 
simply not well aligned with college mathematics 
courses. It may be that NAEP frameworks and items 
are more general and less applied as a reflection of the 
more general nature of high school mathematics and 
the problems students encounter in those classes. 

The NAEP framework sets the parameters for what 
will be included on an assessment; it is not a 
curriculum framework. For this reason, the NAEP 
framework will be less likely than a curriculum 
framework to include the full range of KSAs that are 
prerequisites to at least some entry-level college 
courses (e.g., use of calculators, facility in an online 
environment, ability to solve certain types of 


problems that require applications to real-world 
contexts specific to a disciplinary area). This 
distinction is not unique to the NAEP, but it is 
important to recognize in the area of mathematics 
because of the range of courses offered and their 
connection to the specific academic disciplines in 
which students will apply the mathematics they take 
in an entry-level math course. 

Research Question 3 

How do these prerequisite KSAs compare with previous 
NAEP preparedness research; i.e., the descriptions of minimal 
academic preparedness requirements produced in the JSS 
research ? Compared to the JSS college preparedness 
borderline performance descriptions (BPDs) of KSAs 
necessary for minimal preparedness in entry-level 
college courses, prerequisites identified in the current 
study tend to describe more challenging performance 
in reading. Mathematics prerequisites appear to be 
more nuanced in the current study than is evident in 
the JSS BPD, with prerequisite KSAs differing greatly 
by mathematics course title. 

Reading Prerequisite KSA Comparison With 
Previous NAEP Preparedness Research 
The CCCA prerequisite KSAs describe a depth of 
understanding and a level of cognitive demand that 
are beyond those described by the JSS minimum 
preparedness requirements. The NAEP experts 
described the relationship between the two 
descriptions as concentric circles, with the JSS circle 
completely inside the CCCA circle. They concluded 
that the CCCA study, with the inclusion of the 
courses that relied heavily on informational texts as 
well as those relying primarily on literary texts, 
resulted in a narrative description that was broader 
and deeper than the JSS BPD. 

The reading prerequisites identified in the current 
study are more numerous and constitute more higher- 
order thinking skills than those described by the JSS 


description of minimal preparedness. The specific 
NAEP objectives necessary for minimal preparedness 
from JSS were all subsumed by the prerequisites 
evident in CCCA course artifacts. For example, the 
JSS description included offer evidence in support of, but 
not judging, evaluating, and critiquing, which 
subsumes offering evidence to support a claim. 

Mathematics Prerequisite KSA Comparison With 
Previous NAEP Preparedness Research 
The prerequisites identified for precalculus/ calculus 
and college algebra are similar to the JSS description 
of the KSAs necessary for minimal preparedness for 
entry-level college mathematics courses. The 
prerequisites for finite mathematics and statistics, 
however, require fewer and less cognitively complex 
KSAs than are described by the JSS description. 

The NAEP mathematics experts described the CCCA 
prerequisites as a subset of the JSS borderline 
performance description (BPD). They concluded that 
the JSS BPD was broader than the CCCA prerequisite 
description for precalculus/calculus. Some of the 
differences between the JSS and CCCA studies are 
likely to contribute to the observed differences. For 
example: 

• In the JSS research, the NAEP experts were 
heavily involved in helping the JSS panelists 
understand and interpret the NAEP 
frameworks. In the current study, the NAEP 
experts were available to answer questions 
about the NAEP framework but were 
intentionally less involved because the design 
called for content experts to classify specific 
instances of evidence. 

• Proportional reasoning and estimation are two 
big ideas that may not have been apparent to 
CCCA reviewers. Proportional reasoning is a 
big idea that undergirds all four mathematics 
courses. Instmctors reflecting on a course 


they have taught can see this, but for 
reviewers looking at course artifacts from a 
sample of courses taught by others, 
proportional reasoning may be harder to find. 

• Some prerequisites may not appear until 
weeks into the class, and thus would not be 
observable in the artifacts selected for analysis 
in the CCCA study. This may be especially 
relevant to finite mathematics courses, in 
which the topics of study are not sequential 
and where the first chapter of a text may or 
may not be the best representation of course 
prerequisites. 

Given the differences in evidence, process, and unit of 
analysis, dissimilarities between studies were expected. 
This study identified prerequisite KSAs from the 
evidence contained in college course packets of 
artifacts. The JSS study identified what minimally 
prepared college students need to know and be able to 
do to succeed in entry-level college mathematics 
courses and courses with substantial reading demands. 

Research Question 4 

How can these prerequisites inform future NAEP 
preparedness research ? This research question addresses 
the next steps that arise from the prerequisite KSAs 
found in this study. The NAEP program of 
preparedness research includes five major types of 
studies. The CCCA study relates to Judgmental 
Standard Setting studies. The type of content analysis 
conducted for CCCA does not constitute a major 
strand of research within the preparedness research 
agenda. However, identifying the KSAs that are 
necessary for college or career preparation 
coursework — the coursework students might 
encounter the year following their performance on 
the 12th grade NAEP — broadens understanding of 
the relevance of the 12th grade NAEP with respect to 
academic preparedness for college. Including a suite 
of content analyses as a separate category of studies. 


or expanding the scope of studies currently 
categorized as Judgmental Standard Setting would be 
one way to expand the preparedness research agenda 
to be more inclusive of the questions that should be 
addressed by the research program overall. 

Prerequisites for English literature courses differed 
from those for the other included courses because 
English literature included primarily literary texts, 
while the other courses included only informational 
texts. A separate study focusing specifically on the 
prerequisites from a broader set of entry-level college 
courses might be useful at identifying differences in 
prerequisites for courses that use a variety of texts 
and text types. Courses of interest might include 
mathematics- and graphics-based social science 
courses, such as economics, and courses that include 
both fiction and nonfiction, such as humanities, 
philosophy, or library and information science 
courses. Focusing on what students need to know to 
be prepared for all entry-level college courses that 
require reading and comprehending various forms 
and types of texts will differentiate the knowledge, 
skills, and abilities necessary for each. This is a goal 
consistent with the Governing Board’s perspective 
that reading skills are utilized and necessary in all 
academic contexts and day-to-day real-world 
activities, not just required for high grades in English 
language arts courses. It may also help the NAEP 
team to determine whether changes in parameters 
regarding reading passage selections need to be made 
to capture a wider range of reading skills necessary for 
entry-level college course success. 

An interesting question to consider might be, given 
the discipline-specific knowledge required in most 
courses, should a generic reading framework like the 
current NAEP framework be used for determining 
the necessary prerequisite KSAs or should 
frameworks and accompanying assessments be 
developed that include some measure of subject 


matter knowledge? An analysis of the various NAEP 
assessments and an exploration of prerequisite KSAs 
needed to perform well on those assessments could 
be an interesting study. 

There are course characteristics that may be 
important to future content analysis studies, 
including, but not limited to, the following: 

• The types of items used for quizzes and 
exams early in the course. The level of skills 
required of a multiple choice vs. short answer 
or constmcted-response item would reflect 
the level of cognitive complexity and 
expectations for entering students. 

• The department providing a course may have 
a moderate to profound influence on the 
prerequisite KSAs students will need when 
they enter the course (e.g., statistics offered by 
the mathematics department versus the 
psychology department or calculus offered in 
business versus mathematics departments). 

• The quantitative reasoning skills expected in a 


full range of entry-level college courses and 
the alignment of those quantitative reasoning 
skills to the 4th, 8th, and 12th grade NAEP 
mathematics frameworks. 

Future content analysis studies should present the 
12th grade objectives beside the subsumed 8th and 
4th grade objectives. Any indication of a 12th grade 
objective as prerequisite would also identify the 
corresponding 4th and 8th grade objectives as 
prerequisite. The mathematics experts suggested that 
the NAEP framework taken as a whole across grades 
would provide a better match to the prerequisite 
KSAs than looking solely at the 12th grade 
framework. Future research could include a content 
analysis study using the NAEP frameworks for all 
grades. Although proficient- or advanced-level 
performance on the 4th or 8th grade NAEP 
assessments does not necessarily indicate readiness 
for postsecondary work, a study of the full NAEP 
frameworks may result in the development of a 
preparedness framework that is more inclusive of all 
prerequisite KSAs. 
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